Merge insert has always taken a list of columns to match on, and local
tables have always joined on all of them. Remote tables did not: any
list longer than one was rejected with `MergeInsertBuilder only supports
a single 'on' column`, so a composite-key upsert was impossible against
LanceDB Cloud and Enterprise from Rust, Python or TypeScript.
The remote request now carries `on` as a list and sends it as one
repeated query parameter per column — `?on=shard_key&on=id`. That is how
the lance-namespace spec encodes an array-valued `on`, so the server
receives a composite key in the shape it expects. A single column still
serializes to `?on=id`, exactly what clients sent before, so existing
callers are unaffected. A column repeated within `on` is now rejected
client-side rather than sent for the server to reject with a 400.
No binding changes were needed: `Table.merge_insert` in Python and
`Table.mergeInsert` in TypeScript already accepted a list, it just could
not reach a remote table. Both gain a test for composite keys, and the
doc comments now say what passing several columns means.
Part of
[ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the).
## Example
```python
table.merge_insert(["shard_key", "id"]) \
.when_matched_update_all() \
.when_not_matched_insert_all() \
.execute(new_data)
```
A row whose `id` matches an existing row but whose `shard_key` differs
is an insert, not an update.
## Not included
Java. Java callers reach merge insert through
`org.lance.namespace.LanceNamespace`, whose
`MergeInsertIntoTableRequest.on` is a single string until
lance-namespace 0.12
([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363),
[lance#8915](https://github.com/lance-format/lance/pull/8915)). There is
nothing in this repo's Java SDK to change until the `lance-core` pin can
move.
Sending more than one column requires a server that accepts the repeated
parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571));
an older server returns a 400 rather than silently merging on one
column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remote create-index requests already expose `replace` on the builder,
but the remote client did not consistently forward an explicit
`replace=false` over REST. That meant create-only intent could be lost
before it reached a remote server, even though local builders and Python
APIs can express it. This PR forwards `replace=false` on the existing
`create_index` endpoint and keeps the current default behavior unchanged
for compatibility.
This was accomplished with the following changes:
- Serialize `replace: false` into the existing remote create-index
request body when the builder is configured with `.replace(false)`.
- Forward `replace` through the synchronous Python remote `create_index`
wrapper so `RemoteTable.create_index(..., replace=False)` reaches the
repaired path.
- Continue omitting `replace` for the default path so existing remote
create-index requests keep their current semantics.
- Document `name` and `replace` on the existing OpenAPI create-index
request schema.
- Add coverage that verifies the remote client uses the existing
`/create_index/` route and forwards `replace=false`, including the
synchronous Python unified API.
### Testing
- `cargo fmt --all --check`
- `cargo test -p lancedb --features remote
test_create_index_forwards_replace_false_on_existing_route --locked`
- `uv tool run maturin develop --extras tests,dev,embeddings`
- `uv run --frozen pytest
python/tests/test_remote_db.py::test_remote_create_index_new_api`
- `uv run ruff format --check python/lancedb/remote/table.py
python/tests/test_remote_db.py`
- `cargo build -p lancedb --features remote --locked`
- `cargo clippy -p lancedb --features remote --all-targets --locked --
-D warnings`
Function signatures currently reject Blob v2 fields nested inside
structs, preventing UDFs from accepting or returning structured values
that contain blobs.
Accept canonical Blob v2 fields as direct or recursive struct children
while preserving exact field metadata and nullability. Blob fields under
list, large-list, fixed-size-list, or map ancestors remain rejected
because collection runtime adaptation is outside the supported Function
ABI.
A whole named struct result can bind directly to one destination column
without introducing an extra wrapper level.
Function registration and exact lookup are exposed through the SDK, but
clients cannot discover published versions even though the server
provides `POST /v1/functions/list`.
Add Rust and Python sync/async `list_functions()` APIs that return typed
`FunctionVersion` values. The remote client requests canonical
definitions and follows opaque page tokens until the listing is
complete, including empty intermediate pages, while preserving the
server's name/version ordering. Local databases retain the existing
Function-catalog unsupported error.
The SDK consumes protocol pagination internally so callers receive the
complete catalog rather than handling server-specific page tokens.
A view definition recorded its source by bare name and refresh resolved
that name at the root, so declaring a view over a namespaced source was
refused outright -- materialized views were root-only for every caller.
The definition now carries `source_namespace`, and refresh opens the
source at that coordinate. `plan` takes the namespace too: refresh
re-plans the stored definition and persists the result when it migrates,
so defaulting it there would strand the view on its next rebuild.
The stored kind is the version boundary. Root definitions keep the
`select` form byte-for-byte, so everything written before this change
reads exactly as it always did. A namespaced source is stored as
`namespaced_select`: released readers drop unknown fields and resolve a
`select` source at the root, so keeping the old kind would let a
rolled-back worker refresh a view from a same-name root table -- the new
kind routes them to their existing unrecognized-kind refusal instead.
The Python and Node definition parsers learn the new kind alongside the
Rust core.
## Summary
- preserve repeated table offsets without adding a public ordering
guarantee
- retain exact requested ordering in identity and persisted permutations
- cover local, projected, multi-batch, and mocked-remote query paths
## Root cause
Take queries lowered offsets to a set-like IN predicate and discarded
repeated occurrences. Persisted permutation loading also compared the
distinct base-table result count with the requested occurrence count,
rejecting repeated row IDs before its existing reordering step could
expand them.
## Fix
The shared take-query path now deduplicates the predicate for efficient
lookup, requests row-offset metadata internally, and expands each
matching row to the requested multiplicity in backend result order. An
internal opt-in keeps exact requested order for identity
PermutationReader reads, while persisted permutations continue using
their existing ordering map.
## Validation
- cargo test --quiet --features remote --tests
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
- targeted Python local and mocked-remote regression tests
- exact issue reproduction
Fixes#2820
<!-- lance-gatekeeper-fix:v1 agent=75acf840afa6f4be4bff98b567b504bd
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Other changes
### What changed?
- Add a subprocess regression harness for an ordinary `@udf` function
defined in `__main__`.
- Verify the full registration request, artifact digest, and Function
signature stay identical across independent Python processes and
renamed/moved script paths.
- Verify body, referenced-global, and annotation changes still produce
distinct artifact identities, with annotation changes also producing a
distinct Function signature.
### Why is the change needed?
[ENT-2441](https://linear.app/lancedb/issue/ENT-2441/make-sure-function-defined-in-main-gets-stable-signature)
tracks the stability guarantee. Investigation on the exact `840e1d73`
main base found that LanceDB already packages canonical source instead
of cloudpickle bytes, so the unchanged `__main__` function is stable and
no production-code fix is needed. This change closes the missing
regression-test coverage.
[GEN-950](https://linear.app/lancedb/issue/GEN-950/class-based-udfs-defined-in-main-get-a-new-auto-version-on-every-run)
remains a separate Geneva checkpoint-version issue for class-based
callables. LanceDB's Function API continues to accept synchronous Python
functions only.
## Validation
- `cd python && uv run --extra tests pytest
python/tests/test_first_class_function_slice2.py -q` (`40 passed`)
- `uv run --project python --extra dev ruff format .`
- `uv run --project python --extra dev ruff check .` (`All checks
passed!`)
Teach the Python Function signature emitter to serialize PyArrow large
strings as the canonical Arrow type name `large_utf8`.
Extend the shared Function Arrow type fixture and explicit-schema
coverage for scalar, nested, list, and large-list compositions.
Make Function authoring and declaration planning treat Blob v2 as a
scalar semantic type while preserving exact Blob metadata in binding
schemas.
Covers scalar Blob outputs, expanded named-struct outputs, and
whole-result structs with Blob children.
Functions can describe their Python environment today, but cannot
declare accelerator requirements. That prevents Sophon from scheduling
computed-column UDF refreshes onto GPU workers from the immutable
Function definition.
Add `num_gpus` to Python `@udf` through a typed
`FunctionResourceRequirements` value and represent resource-aware
definitions with the `python_v2` runtime discriminator. CPU Functions
retain their existing `python` encoding and canonical identity.
The new discriminator is intentional for mixed-version safety:
deployments that do not understand execution resources reject the
runtime instead of accepting a new field and silently running the
Function on CPU. Required resources are part of Function version
identity; priority, concurrency, and retry policy remain Job concerns.
The actual resource scheduling remains owned by Sophon.
Teach Python Function authoring to retain the compact V1 grammar for
existing types and emit canonical exact JSON for nested struct
signatures. Adds coverage for recursive struct/list schemas and exact
field properties.
Computed-column planning currently sees Blob v2 storage descriptors, so
expressions cannot consume payload bytes or preserve Blob semantics in
their outputs.
A computed declaration now derives its output field from its expression.
A direct projection of a Blob v2 field inherits the source field's Blob
metadata; other expressions retain their ordinary Arrow-inferred type.
Declarations remain ordered, so the same rule applies across chained
projections.
Refresh materializes referenced Blob inputs as `LargeBinary` payload
bytes and publishes inherited Blob outputs through Lance's Blob
conversion path. Remote requests remain within the shared namespace
contract as `{name, computed}`; the server planner is being updated in
tandem to implement the same Blob-aware planning semantics, and remote
enablement must be aligned with that server rollout.
The existing null-as-unfilled contract remains unchanged. Row-level
freshness and cell flags remain follow-up work.
## Summary
- allow Python sync, async, and remote table updates to accept type-safe
`Expr` filters
- serialize expression filters before invoking the existing update
implementation
- cover numeric-looking text and apostrophe-containing text in sync and
async regression tests
## Root cause
`Table.update` was the remaining Python write path that required callers
to construct a raw SQL predicate. Dynamic text interpolated without SQL
literal encoding could therefore be parsed as an integer, float, or
unterminated string instead of Utf8. The expression API already encodes
literals safely for query and delete filters.
## Validation
- `cd python && .venv/bin/pytest
python/tests/test_table.py::test_update_async
python/tests/test_table.py::test_update_expr_filter_literals_async
python/tests/test_table.py::test_update
python/tests/test_table.py::test_update_expr_filter_literals -q`
- `cd python && .venv/bin/pytest python/tests/test_expr.py -q`
- `cd python && .venv/bin/ruff format --check .`
- `cd python && .venv/bin/ruff check .`
Fixes#1869
<!-- lance-gatekeeper-fix:v1 agent=01f1e7b69c65e8b6d3b3c1e1a7918179
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- add Python regression coverage for integer and double arithmetic
against the generated _distance column
- merge the current main base containing Lance v11.0.0-beta.3 from #3896
- verify both expressions retain the generated scoring field Float32
type and compute the expected values
## Root cause
Lance parsed dynamic projection expressions before vector search added
its generated Float32 _distance field. Without a typed provisional
field, expression discovery rejected mixed numeric arithmetic. Lance
upstream fixed discovery and final-schema replanning in
lance-format/lance#8163, and the current base consumes that fix through
Lance v11.0.0-beta.3.
## Validation
- uv run --extra tests pytest
python/tests/test_query.py::test_select_arithmetic_with_distance -vv
--maxfail=2 — 2 passed
- python/.venv/bin/ruff format --check python/python/tests/test_query.py
- python/.venv/bin/ruff check .
Fixes#2618
<!-- lance-gatekeeper-fix:v1 agent=816a060090517471edfb73652bb5c9fe
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
<!-- lance-gatekeeper-fix:v1 agent=d30696bc46eb32f04c9927b0792e35d3
generation=1 -->
## Summary
- use the Lance native batch KNN path so fixed-size batch vector
searches share one flat table scan
- validate consistent query-vector dimensions and retain the per-vector
plan when offsets require its existing semantics
- add Rust and Python regressions and update Rust, Python, and
TypeScript API documentation
## Root cause
LanceDB expanded every vector in a batch into a separate scan plan and
joined the plans with `UnionExec`. For unindexed tables on S3, a batch
of ten vectors therefore ran ten concurrent full scans, amplifying CPU
and retained data enough to produce the reported memory spike.
The native Lance batch KNN path performs bounded-memory selection for
all query vectors over one flat scan. LanceDB now supplies the vectors
as a batch and avoids applying a global scanner limit to the combined
per-query results. Batch queries with a nonzero offset keep the previous
plan because the native batch API does not support per-query offsets.
## Validation
- targeted Rust batch-query plan and execution tests
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo fmt --all -- --check`
- targeted Python batch-vector regression after rebuilding the extension
- Ruff formatting/checks for the touched Python files
- Node.js build, lint, docs generation, and targeted batch-vector Jest
test
- `git diff --check`
Fixes#2468
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
In LanceDB Enterprise, we've adopted these conventions to give some
"canonical" metadata paths. This lets us display them in a certain way
in the UI or let agents standardize on them, to assume they'll find info
in a certain place. This PR (only comments/docs) just documents those
choices.
## Summary
LanceDB could not request Lance's list-element FTS document granularity
through Python or Remote APIs, and generic nested-field resolution
exposed Arrow's internal `item` segment instead of the public field
path.
This exposes typed `row | list_element` configuration for Python FTS
index creation and match/phrase queries, preserves `_doc_index`, and
keeps nested FTS paths public (for example, `docs.content`). Remote
list-element requests require server API version 0.6.0 so older servers
cannot silently execute them with row semantics; explicit row requests
remain compatible.
## Compatibility
Omitted index and query parameters retain row granularity. Remote row
index creation omits the new wire field.
## Tracking
[ENT-2342](https://linear.app/lancedb/issue/ENT-2342/expose-list-element-fts-document-granularity-end-to-end)
A Function's remote environment can now be conda instead of pip.
`@udf(conda=[...], conda_channels=[...])` registers one; pip and conda
are exclusive, channels are priority-ordered and require conda. The Rust
and Python `PythonEnvironmentSpec` models gain `channels`, dropped from
the canonical JSON when empty so existing pip registrations keep their
digests.
## Summary
- add regression coverage for the reported synchronous Python workload
with 32 simultaneous `open_table` calls
- verify every independently opened S3-backed table handle can read
through the connection's shared session and object-store client
## Root cause
In Python v0.13.0, each synchronous table handle lazily constructed its
own Lance dataset. Opening many handles in parallel therefore triggered
independent S3 client construction and bucket-region resolution, which
failed under thread pressure. The current Rust-backed connection path
owns a shared Lance session and retains its object-store handle, so
table opens reuse the existing S3 client; these tests lock in that
behavior through the public Python API and a causal Session-registry
invariant.
## Validation
- `uvx --from 'ruff==0.15.20' ruff format --check
python/tests/test_s3.py`
- `uvx --from 'ruff==0.15.20' ruff check .`
- `cargo fmt --all`
- `cargo test --quiet --features remote -p lancedb
test_concurrent_open_table_reuses_connection_object_store`
- `cargo check --quiet --features remote --tests --examples`
- equivalent 32-thread `open_table(...).count_rows()` workload against a
local database
- targeted S3 test collected successfully locally; execution requires
the CI LocalStack service, which is unavailable in this runner
Fixes#1786
<!-- lance-gatekeeper-fix:v1 agent=d311f3c7151f77ae22b4997702e7b7db
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- document that Azure Blob Storage credentials can be passed directly
through `storage_options`
- provide valid quoted `account_name` and `account_key` examples for
both sync and async Python connections
- execute the option dictionaries during doctests so the original
unquoted-key mistake is caught
## Root cause
The historical Python storage guide used `account_name` and
`account_key` as bare identifiers in dictionary literals. Following that
example either raised `NameError` or, when those names were predefined,
produced incorrect option keys. The runtime already accepts direct Azure
credentials, but the current Python API reference did not contain a
corrected Azure example.
## Validation
- `python/.venv/bin/ruff format --check
python/python/lancedb/__init__.py`
- `python/.venv/bin/ruff check .`
- `cd python && uv run --no-sync pytest --doctest-modules
python/lancedb/__init__.py -q`
- `cd python && uv run --no-sync pytest python/tests/test_import.py -q`
Fixes#2236
<!-- lance-gatekeeper-fix:v1 agent=7a7a9e009eb2ebe52ac9b1adf2e8afb2
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
#3528 added blob declarations and binary coercion. String values were
still rejected. They now coerce to the blob `uri` child.
```python
table.add([{"id": 1, "image": "s3://bucket/media/cat.jpg"}])
payload = table.fetch_blobs("image", table.search().to_arrow())
```
A URI under a registered base writes with no extra options. An
unregistered URI fails. `allow_external_blob_outside_bases` is a local
escape hatch that stores an absolute URI. It does not register a base.
Remote `add` rejects that flag before making a request. String input
still coerces and is sent as a `uri` struct.
`add_bases` is a follow-up. `merge_insert` does not coerce string blob
input.
### Testing
- `cargo test -p lancedb --test blob_integration`
- `cargo test -p lancedb blob_coerce`
- `cargo test -p lancedb --features remote --lib
add_rejects_external_blob_flag add_string_blob_becomes_uri_struct`
- `cd python && uv run --extra tests pytest python/tests/test_blob.py -k
uri -q`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Problem
`uv run ... maturin develop` synchronizes the project as an editable
package before running the command. Maturin's editable build otherwise
uses the release profile, which enables the repository's fat LTO
configuration during local bootstrap.
## Behavior
Editable Python builds now explicitly use Cargo's dev profile. The
minimum maturin version is raised to 1.10, where `editable-profile`
support was introduced.
## Problem
The unreleased First-Class Function authoring API exposed
`secrets=[...]` and serialized `required_secrets`, promising runtime
resolution and injection that Sophon does not implement.
## Behavior
Remove the secrets dimension from the public Python decorator, Python
and Rust registration/version models, and shared wire fixtures. Existing
successfully registered Functions retain stable identity: the field
could only be empty and empty values were already omitted from canonical
serialization.
A stable LanceDB release has not shipped this Function API, so this
contracts the surface before it becomes a published compatibility
commitment.
## Validation gap
The Rust shared-golden Function suites and Python formatting/lint checks
pass locally. Python pytest was not run because the local environment
lacks its runtime dependencies and the frozen native editable build did
not complete in practical time.
Function applications and bindings currently encode `group_id` and
binding `revision` even though `binding_id` already owns the complete
immutable binding lifecycle and `outputs` already defines the atomic
multi-output set.
Make `binding_id` the sole binding identity, remove the redundant fields
from the Rust and Python client contracts, and describe multi-output
declarations directly. This intentionally replaces the removed wire
fields without a compatibility path.
Registering a real (embedding) Function failed on the client for three
reasons:
- `_package_source` treated `inspect.getclosurevars().unbound` as
"unresolved globals"; CPython puts attribute names there, so any body
with `np.linalg.norm(...)` or `body.split()` was rejected. Module-scope
references now come from Python's own scope analysis (`symtable`) over
the function source, recursively, and each is resolved the way the
interpreter would: the function's globals first (a module global may
shadow a builtin), then builtins. Free variables of nested scopes stay
lexical; postponed annotations are not runtime loads. A genuinely
missing global still fails.
- `_canonical_arrow_type` emitted spellings the server's frozen grammar
rejects (`fixed_size_list<T>[n]`, `timestamp[us]`, `struct<...>`,
zero-sized lists). It now emits exactly the grammar, with the server's
`fixed_size_list<item, size>` form, and the Rust declaration planner
parses that form too.
A shared golden
(`tests/fixtures/first_class_functions/v1/arrow_types.json`)
enumerates every grammar type, nested forms and rejected spellings; the
Python emitter and Rust parser are tested against it, and the same file
is under test in sophon. Packaging tests execute the shipped artifact in
a fresh namespace.
Contract changes (hence `breaking-change`):
- `@udf` now rejects namespace acquisition structurally
(`globals()`/`eval`/... by name, plus `import
sys`/`builtins`/`importlib`/`inspect` inside the body), requires the
function's captured `__builtins__` to be the standard mapping itself
(identity, so neither lookups nor implicit hooks such as `__import__`
can differ), rejects module globals that are namespace-bearing modules
(`builtins`, `sys`, ...), and treats the function's own name as
recursion only when the module binds it to the function or to the exact
`UdfDefinition` the decorator produced; it resolves module globals
through the function's real namespace (a module global may shadow a
builtin) and ships importable classes/functions as imports.
- List outputs must declare a non-nullable, metadata-free child named
`item` (`pa.list_(pa.field("item", t, nullable=False))`); that is what
the grammar means, and pyarrow's default nullable child was being
silently collapsed into it.
Contract, stated in the `udf` docstring: the artifact is a snapshot of
the function source plus exactly the module names it references.
Reaching the module namespace by another route is rejected where a
static packager can see it and is otherwise unsupported; there is no
dynamic-access detection beyond that.
A permutation stores `_rowid`s, which are row addresses unless stable
row ids are enabled. Nothing in the data loader pinned a table version,
so a compaction between building a permutation and reading it can
resolve those ids to different rows.
The exposure differs by backend but exists on both:
- Remote never pins. `prepare_query_bodies` stamps `"version":
current_version()` on every request, but `current_version()` is `None`
unless `checkout` was called, so every request means "latest".
- Native pins implicitly by holding an `Arc<Dataset>` under
`ConsistencyMode::Lazy`, but `StreamingDataset.__setstate__` reopens the
table in each DataLoader worker, so each worker pins to whatever is
latest at fork time.
## Changes
`Table::at_version` returns an independent handle pinned to a version
without mutating the receiver. `checkout` cannot serve this: on remote
the version cell is an `Arc<RwLock<Option<u64>>>` shared across clones,
so pinning through it would silently pin the caller's table too.
`PermutationBuilder::build` pins for the whole build and records the
version in the permutation table's schema metadata, alongside the
existing split names. `PermutationReader` pins the base table to that
version before any take.
Because the reader pins on construction, the Python worker fork is
covered without touching the pickle format — `Permutation.__setstate__`
drops the reader and `_ensure_open` rebuilds it, which re-pins.
## Behaviour change
A permutation is now bound to the version it was built against, so rows
appended to the base table afterwards are not visible through an
existing permutation. That is the intended semantics — the permutation
only addresses rows that existed when it was built — but it is a change
worth flagging.
Permutations written before this carry no version key and read exactly
as they did before.
## Summary
- add `StreamingDataLoader`, which transports worker snapshots with
prefetched batches and commits them to the parent dataset only when the
trainer receives each batch
- preserve exact non-uniform per-split progress and resume lagging
splits without replaying already-consumed rows
- reject stale parent checkpoints after a standard multi-process
`DataLoader` has started, with guidance to use the consumer-aware loader
- document the new public loader and merge non-uniform state across
ranks
## Root cause
PyTorch runs `StreamingDataset.__iter__` in private worker-process
copies, while callers invoke `state_dict()` on the parent dataset.
Sharing producer counters would still be incorrect because DataLoader
prefetch can advance workers beyond batches returned to the trainer.
## Validation
- `uv run --extra tests pytest python/tests/test_elastic_dataloader.py
-q` (154 passed)
- focused non-uniform merge regression (1 passed)
- `uv run --project python --extra tests --extra dev ruff format .`
- `uv run --project python --extra tests --extra dev ruff check .`
- `cd docs && PYTHONPATH=. ../python/.venv/bin/mkdocs build`
Fixes#3967
<!-- lance-gatekeeper-fix:v1 agent=572be272619660b97e87fd5c85188341
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Problem
`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.
## Behavior
`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.
The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.
## Ownership and integration boundary
LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
## How packing works
Consider four tokenized documents:
[1]
[2]
[10, 11, 12, 13, 14, 15, 16, 17]
[20]
With:
```
StreamingDataset(
table,
shuffle=False,
columns=["tokens"],
num_splits=2,
pack_sequences=5,
eos_id=9,
pad_id=0,
blocks_per_epoch=6,
)
```
the documents are assigned to two fixed logical splits. Each split
maintains an independent token buffer, appends eos_id after every
document, and emits blocks of five tokens.
Because blocks_per_epoch=6, each split emits exactly three blocks:
Cycl/e 1:
Split 0: [1, 9, 2, 9, 0] # 9 is eos, 0 is padding
Split 1: [10, 11, 12, 13, 14]
Cycle 2:
Split 0: [0, 0, 0, 0, 0]
Split 1: [15, 16, 17, 9, 20]
Cycle 3:
Split 0: [0, 0, 0, 0, 0]
Split 1: [9, 0, 0, 0, 0]
If a split runs out of tokens early, it emits padded blocks through the
fixed budget. This prevents one rank from finishing before another.
Logical splits are independent of rank and worker ownership. A
checkpoint records each split’s consumed-document count, emitted-block
count, remaining tokens, and document boundaries. Merging
those per-split states allows the same packed stream to resume after the
topology changes.
doc_ids identifies document segments, including continuations across
block boundaries. It is not a padding mask: padding retains the
preceding document ID, so callers must mask padding using a
reserved pad_id.
blocks_per_epoch="auto" is also available. It estimates the budget from
a deterministic bounded sample and warns that the result is approximate.
WIP pre-training tests:
```
┌────────────────────────────────────┬────────────────────────┬─────────────────────────────────────┐
│ │ GPT-2 124M │ GPT-2 medium 354M │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Corpus │ 2.4M docs / 12GB table │ 9.67M docs / 45GB table │
│ Tokens (Chinchilla) │ 2.43B │ 7.0B │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Data prep (ingest→curate→tokenize) │ ~12 min │ ~51 min │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Training wall time │ ~50 min │ 3h 06m │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Throughput / MFU │ 1.60M tok/s / 35% │ 684k tok/s / 42.0%, │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Final val loss │ 3.230 │ 2.840 │
└────────────────────────────────────┴────────────────────────┴─────────────────────────────────────┘
```
---------
Co-authored-by: OpenAI Codex <codex@openai.com>
Rename prefetch_batches → io_queue_depth and introduce
transform_queue_depth as a symmetric pair: both express "number of
batches to buffer per split at this pipeline stage." The old names are
still accepted as keyword arguments but log a deprecation warning
redirecting callers to the new names.
transform_queue_depth caps how many transform-result batches can
accumulate per split in the post-transform queue. Without this limit a
slow consumer (e.g. a GPU training step) causes cooked rows to pile up
unboundedly. The backpressure check in _try_submit_tx counts both
already-cooked rows and rows expected from in-flight transforms; it
skips proactive transform submission when the combined total reaches the
limit. The reactive _ensure_cooked path bypasses the check so the
consumer never stalls.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This PR is a **breaking** rename of #3686.
merge reads like git merge w/ three-way, replay history, combine two
lines of work. That is not this API.
This call takes one additive change on a branch and lands it on main.
New column, including a blob column. Main's existing columns are not
rewritten. If it cannot land, you get `status="failed"` and
`diff.errors`, not a merge conflict to resolve.
Cherry-pick is terminology that aligns more with that.
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.cherry_pick("exp", dry_run=True)
result = table.branches.cherry_pick("exp")
if result["status"] == "cherryPicked":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "failed":
print(result["diff"]["errors"])
```
### Behavior
- Remote / Enterprise only. Local still NotSupported.
- HTTP 409 is not an exception. It is Ok with status="failed" and
diff.errors (CherryPickError).
- Unknown error / status codes still parse as Unknown.
- Requests are not retried. 409 is final and carries the body.
- Endpoint is POST /v1/table/{id}/branches/cherry_pick/.
- merge_insert and Table.merge are unchanged.
### Testing
- `cargo test -p lancedb --features remote diff_branch`
- `cargo test -p lancedb --features remote cherry_pick`
- `pytest python/python/tests/test_remote_db.py -k cherry_pick`
- node `remote.test.ts` diffs / cherry-picks path
Exposes materialized views to Python in both the async and sync clients:
create_materialized_view / open_materialized_view /
list_materialized_views
on the connections, and MaterializedView / AsyncMaterializedView handles
carrying the parsed definition and refresh(full=, source_version=),
which
returns the typed refresh result. select accepts column names, (alias,
expression) pairs, or a dict of the same; the definition reads back off
the
stored schema, so a reopened handle needs no side channel. Remote
connections raise NotImplementedError up front rather than failing deep
in
a request, matching the computed-column convention.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
Will Jones
Bruno Ramirez
Xuanwo
Lance Release
Wyatt Alt
lancedb-gatefixer[bot]
Yang Cen
Jack Ye
Drew
Dan Tasse
Ayush Chaurasia
Weston Pace