<!-- lance-gatekeeper-fix:v1 agent=d30696bc46eb32f04c9927b0792e35d3
generation=1 -->
## Summary
- use the Lance native batch KNN path so fixed-size batch vector
searches share one flat table scan
- validate consistent query-vector dimensions and retain the per-vector
plan when offsets require its existing semantics
- add Rust and Python regressions and update Rust, Python, and
TypeScript API documentation
## Root cause
LanceDB expanded every vector in a batch into a separate scan plan and
joined the plans with `UnionExec`. For unindexed tables on S3, a batch
of ten vectors therefore ran ten concurrent full scans, amplifying CPU
and retained data enough to produce the reported memory spike.
The native Lance batch KNN path performs bounded-memory selection for
all query vectors over one flat scan. LanceDB now supplies the vectors
as a batch and avoids applying a global scanner limit to the combined
per-query results. Batch queries with a nonzero offset keep the previous
plan because the native batch API does not support per-query offsets.
## Validation
- targeted Rust batch-query plan and execution tests
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo fmt --all -- --check`
- targeted Python batch-vector regression after rebuilding the extension
- Ruff formatting/checks for the touched Python files
- Node.js build, lint, docs generation, and targeted batch-vector Jest
test
- `git diff --check`
Fixes#2468
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
In LanceDB Enterprise, we've adopted these conventions to give some
"canonical" metadata paths. This lets us display them in a certain way
in the UI or let agents standardize on them, to assume they'll find info
in a certain place. This PR (only comments/docs) just documents those
choices.
## Summary
LanceDB could not request Lance's list-element FTS document granularity
through Python or Remote APIs, and generic nested-field resolution
exposed Arrow's internal `item` segment instead of the public field
path.
This exposes typed `row | list_element` configuration for Python FTS
index creation and match/phrase queries, preserves `_doc_index`, and
keeps nested FTS paths public (for example, `docs.content`). Remote
list-element requests require server API version 0.6.0 so older servers
cannot silently execute them with row semantics; explicit row requests
remain compatible.
## Compatibility
Omitted index and query parameters retain row granularity. Remote row
index creation omits the new wire field.
## Tracking
[ENT-2342](https://linear.app/lancedb/issue/ENT-2342/expose-list-element-fts-document-granularity-end-to-end)
A Function's remote environment can now be conda instead of pip.
`@udf(conda=[...], conda_channels=[...])` registers one; pip and conda
are exclusive, channels are priority-ordered and require conda. The Rust
and Python `PythonEnvironmentSpec` models gain `channels`, dropped from
the canonical JSON when empty so existing pip registrations keep their
digests.
## Summary
- add regression coverage for the reported synchronous Python workload
with 32 simultaneous `open_table` calls
- verify every independently opened S3-backed table handle can read
through the connection's shared session and object-store client
## Root cause
In Python v0.13.0, each synchronous table handle lazily constructed its
own Lance dataset. Opening many handles in parallel therefore triggered
independent S3 client construction and bucket-region resolution, which
failed under thread pressure. The current Rust-backed connection path
owns a shared Lance session and retains its object-store handle, so
table opens reuse the existing S3 client; these tests lock in that
behavior through the public Python API and a causal Session-registry
invariant.
## Validation
- `uvx --from 'ruff==0.15.20' ruff format --check
python/tests/test_s3.py`
- `uvx --from 'ruff==0.15.20' ruff check .`
- `cargo fmt --all`
- `cargo test --quiet --features remote -p lancedb
test_concurrent_open_table_reuses_connection_object_store`
- `cargo check --quiet --features remote --tests --examples`
- equivalent 32-thread `open_table(...).count_rows()` workload against a
local database
- targeted S3 test collected successfully locally; execution requires
the CI LocalStack service, which is unavailable in this runner
Fixes#1786
<!-- lance-gatekeeper-fix:v1 agent=d311f3c7151f77ae22b4997702e7b7db
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- document that Azure Blob Storage credentials can be passed directly
through `storage_options`
- provide valid quoted `account_name` and `account_key` examples for
both sync and async Python connections
- execute the option dictionaries during doctests so the original
unquoted-key mistake is caught
## Root cause
The historical Python storage guide used `account_name` and
`account_key` as bare identifiers in dictionary literals. Following that
example either raised `NameError` or, when those names were predefined,
produced incorrect option keys. The runtime already accepts direct Azure
credentials, but the current Python API reference did not contain a
corrected Azure example.
## Validation
- `python/.venv/bin/ruff format --check
python/python/lancedb/__init__.py`
- `python/.venv/bin/ruff check .`
- `cd python && uv run --no-sync pytest --doctest-modules
python/lancedb/__init__.py -q`
- `cd python && uv run --no-sync pytest python/tests/test_import.py -q`
Fixes#2236
<!-- lance-gatekeeper-fix:v1 agent=7a7a9e009eb2ebe52ac9b1adf2e8afb2
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
#3528 added blob declarations and binary coercion. String values were
still rejected. They now coerce to the blob `uri` child.
```python
table.add([{"id": 1, "image": "s3://bucket/media/cat.jpg"}])
payload = table.fetch_blobs("image", table.search().to_arrow())
```
A URI under a registered base writes with no extra options. An
unregistered URI fails. `allow_external_blob_outside_bases` is a local
escape hatch that stores an absolute URI. It does not register a base.
Remote `add` rejects that flag before making a request. String input
still coerces and is sent as a `uri` struct.
`add_bases` is a follow-up. `merge_insert` does not coerce string blob
input.
### Testing
- `cargo test -p lancedb --test blob_integration`
- `cargo test -p lancedb blob_coerce`
- `cargo test -p lancedb --features remote --lib
add_rejects_external_blob_flag add_string_blob_becomes_uri_struct`
- `cd python && uv run --extra tests pytest python/tests/test_blob.py -k
uri -q`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Problem
`uv run ... maturin develop` synchronizes the project as an editable
package before running the command. Maturin's editable build otherwise
uses the release profile, which enables the repository's fat LTO
configuration during local bootstrap.
## Behavior
Editable Python builds now explicitly use Cargo's dev profile. The
minimum maturin version is raised to 1.10, where `editable-profile`
support was introduced.
## Problem
The unreleased First-Class Function authoring API exposed
`secrets=[...]` and serialized `required_secrets`, promising runtime
resolution and injection that Sophon does not implement.
## Behavior
Remove the secrets dimension from the public Python decorator, Python
and Rust registration/version models, and shared wire fixtures. Existing
successfully registered Functions retain stable identity: the field
could only be empty and empty values were already omitted from canonical
serialization.
A stable LanceDB release has not shipped this Function API, so this
contracts the surface before it becomes a published compatibility
commitment.
## Validation gap
The Rust shared-golden Function suites and Python formatting/lint checks
pass locally. Python pytest was not run because the local environment
lacks its runtime dependencies and the frozen native editable build did
not complete in practical time.
Function applications and bindings currently encode `group_id` and
binding `revision` even though `binding_id` already owns the complete
immutable binding lifecycle and `outputs` already defines the atomic
multi-output set.
Make `binding_id` the sole binding identity, remove the redundant fields
from the Rust and Python client contracts, and describe multi-output
declarations directly. This intentionally replaces the removed wire
fields without a compatibility path.
Registering a real (embedding) Function failed on the client for three
reasons:
- `_package_source` treated `inspect.getclosurevars().unbound` as
"unresolved globals"; CPython puts attribute names there, so any body
with `np.linalg.norm(...)` or `body.split()` was rejected. Module-scope
references now come from Python's own scope analysis (`symtable`) over
the function source, recursively, and each is resolved the way the
interpreter would: the function's globals first (a module global may
shadow a builtin), then builtins. Free variables of nested scopes stay
lexical; postponed annotations are not runtime loads. A genuinely
missing global still fails.
- `_canonical_arrow_type` emitted spellings the server's frozen grammar
rejects (`fixed_size_list<T>[n]`, `timestamp[us]`, `struct<...>`,
zero-sized lists). It now emits exactly the grammar, with the server's
`fixed_size_list<item, size>` form, and the Rust declaration planner
parses that form too.
A shared golden
(`tests/fixtures/first_class_functions/v1/arrow_types.json`)
enumerates every grammar type, nested forms and rejected spellings; the
Python emitter and Rust parser are tested against it, and the same file
is under test in sophon. Packaging tests execute the shipped artifact in
a fresh namespace.
Contract changes (hence `breaking-change`):
- `@udf` now rejects namespace acquisition structurally
(`globals()`/`eval`/... by name, plus `import
sys`/`builtins`/`importlib`/`inspect` inside the body), requires the
function's captured `__builtins__` to be the standard mapping itself
(identity, so neither lookups nor implicit hooks such as `__import__`
can differ), rejects module globals that are namespace-bearing modules
(`builtins`, `sys`, ...), and treats the function's own name as
recursion only when the module binds it to the function or to the exact
`UdfDefinition` the decorator produced; it resolves module globals
through the function's real namespace (a module global may shadow a
builtin) and ships importable classes/functions as imports.
- List outputs must declare a non-nullable, metadata-free child named
`item` (`pa.list_(pa.field("item", t, nullable=False))`); that is what
the grammar means, and pyarrow's default nullable child was being
silently collapsed into it.
Contract, stated in the `udf` docstring: the artifact is a snapshot of
the function source plus exactly the module names it references.
Reaching the module namespace by another route is rejected where a
static packager can see it and is otherwise unsupported; there is no
dynamic-access detection beyond that.
A permutation stores `_rowid`s, which are row addresses unless stable
row ids are enabled. Nothing in the data loader pinned a table version,
so a compaction between building a permutation and reading it can
resolve those ids to different rows.
The exposure differs by backend but exists on both:
- Remote never pins. `prepare_query_bodies` stamps `"version":
current_version()` on every request, but `current_version()` is `None`
unless `checkout` was called, so every request means "latest".
- Native pins implicitly by holding an `Arc<Dataset>` under
`ConsistencyMode::Lazy`, but `StreamingDataset.__setstate__` reopens the
table in each DataLoader worker, so each worker pins to whatever is
latest at fork time.
## Changes
`Table::at_version` returns an independent handle pinned to a version
without mutating the receiver. `checkout` cannot serve this: on remote
the version cell is an `Arc<RwLock<Option<u64>>>` shared across clones,
so pinning through it would silently pin the caller's table too.
`PermutationBuilder::build` pins for the whole build and records the
version in the permutation table's schema metadata, alongside the
existing split names. `PermutationReader` pins the base table to that
version before any take.
Because the reader pins on construction, the Python worker fork is
covered without touching the pickle format — `Permutation.__setstate__`
drops the reader and `_ensure_open` rebuilds it, which re-pins.
## Behaviour change
A permutation is now bound to the version it was built against, so rows
appended to the base table afterwards are not visible through an
existing permutation. That is the intended semantics — the permutation
only addresses rows that existed when it was built — but it is a change
worth flagging.
Permutations written before this carry no version key and read exactly
as they did before.
## Summary
- add `StreamingDataLoader`, which transports worker snapshots with
prefetched batches and commits them to the parent dataset only when the
trainer receives each batch
- preserve exact non-uniform per-split progress and resume lagging
splits without replaying already-consumed rows
- reject stale parent checkpoints after a standard multi-process
`DataLoader` has started, with guidance to use the consumer-aware loader
- document the new public loader and merge non-uniform state across
ranks
## Root cause
PyTorch runs `StreamingDataset.__iter__` in private worker-process
copies, while callers invoke `state_dict()` on the parent dataset.
Sharing producer counters would still be incorrect because DataLoader
prefetch can advance workers beyond batches returned to the trainer.
## Validation
- `uv run --extra tests pytest python/tests/test_elastic_dataloader.py
-q` (154 passed)
- focused non-uniform merge regression (1 passed)
- `uv run --project python --extra tests --extra dev ruff format .`
- `uv run --project python --extra tests --extra dev ruff check .`
- `cd docs && PYTHONPATH=. ../python/.venv/bin/mkdocs build`
Fixes#3967
<!-- lance-gatekeeper-fix:v1 agent=572be272619660b97e87fd5c85188341
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Problem
`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.
## Behavior
`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.
The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.
## Ownership and integration boundary
LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
## How packing works
Consider four tokenized documents:
[1]
[2]
[10, 11, 12, 13, 14, 15, 16, 17]
[20]
With:
```
StreamingDataset(
table,
shuffle=False,
columns=["tokens"],
num_splits=2,
pack_sequences=5,
eos_id=9,
pad_id=0,
blocks_per_epoch=6,
)
```
the documents are assigned to two fixed logical splits. Each split
maintains an independent token buffer, appends eos_id after every
document, and emits blocks of five tokens.
Because blocks_per_epoch=6, each split emits exactly three blocks:
Cycl/e 1:
Split 0: [1, 9, 2, 9, 0] # 9 is eos, 0 is padding
Split 1: [10, 11, 12, 13, 14]
Cycle 2:
Split 0: [0, 0, 0, 0, 0]
Split 1: [15, 16, 17, 9, 20]
Cycle 3:
Split 0: [0, 0, 0, 0, 0]
Split 1: [9, 0, 0, 0, 0]
If a split runs out of tokens early, it emits padded blocks through the
fixed budget. This prevents one rank from finishing before another.
Logical splits are independent of rank and worker ownership. A
checkpoint records each split’s consumed-document count, emitted-block
count, remaining tokens, and document boundaries. Merging
those per-split states allows the same packed stream to resume after the
topology changes.
doc_ids identifies document segments, including continuations across
block boundaries. It is not a padding mask: padding retains the
preceding document ID, so callers must mask padding using a
reserved pad_id.
blocks_per_epoch="auto" is also available. It estimates the budget from
a deterministic bounded sample and warns that the result is approximate.
WIP pre-training tests:
```
┌────────────────────────────────────┬────────────────────────┬─────────────────────────────────────┐
│ │ GPT-2 124M │ GPT-2 medium 354M │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Corpus │ 2.4M docs / 12GB table │ 9.67M docs / 45GB table │
│ Tokens (Chinchilla) │ 2.43B │ 7.0B │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Data prep (ingest→curate→tokenize) │ ~12 min │ ~51 min │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Training wall time │ ~50 min │ 3h 06m │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Throughput / MFU │ 1.60M tok/s / 35% │ 684k tok/s / 42.0%, │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Final val loss │ 3.230 │ 2.840 │
└────────────────────────────────────┴────────────────────────┴─────────────────────────────────────┘
```
---------
Co-authored-by: OpenAI Codex <codex@openai.com>
Rename prefetch_batches → io_queue_depth and introduce
transform_queue_depth as a symmetric pair: both express "number of
batches to buffer per split at this pipeline stage." The old names are
still accepted as keyword arguments but log a deprecation warning
redirecting callers to the new names.
transform_queue_depth caps how many transform-result batches can
accumulate per split in the post-transform queue. Without this limit a
slow consumer (e.g. a GPU training step) causes cooked rows to pile up
unboundedly. The backpressure check in _try_submit_tx counts both
already-cooked rows and rows expected from in-flight transforms; it
skips proactive transform submission when the combined total reaches the
limit. The reactive _ensure_cooked path bypasses the check so the
consumer never stalls.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This PR is a **breaking** rename of #3686.
merge reads like git merge w/ three-way, replay history, combine two
lines of work. That is not this API.
This call takes one additive change on a branch and lands it on main.
New column, including a blob column. Main's existing columns are not
rewritten. If it cannot land, you get `status="failed"` and
`diff.errors`, not a merge conflict to resolve.
Cherry-pick is terminology that aligns more with that.
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.cherry_pick("exp", dry_run=True)
result = table.branches.cherry_pick("exp")
if result["status"] == "cherryPicked":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "failed":
print(result["diff"]["errors"])
```
### Behavior
- Remote / Enterprise only. Local still NotSupported.
- HTTP 409 is not an exception. It is Ok with status="failed" and
diff.errors (CherryPickError).
- Unknown error / status codes still parse as Unknown.
- Requests are not retried. 409 is final and carries the body.
- Endpoint is POST /v1/table/{id}/branches/cherry_pick/.
- merge_insert and Table.merge are unchanged.
### Testing
- `cargo test -p lancedb --features remote diff_branch`
- `cargo test -p lancedb --features remote cherry_pick`
- `pytest python/python/tests/test_remote_db.py -k cherry_pick`
- node `remote.test.ts` diffs / cherry-picks path
Exposes materialized views to Python in both the async and sync clients:
create_materialized_view / open_materialized_view /
list_materialized_views
on the connections, and MaterializedView / AsyncMaterializedView handles
carrying the parsed definition and refresh(full=, source_version=),
which
returns the typed refresh result. select accepts column names, (alias,
expression) pairs, or a dict of the same; the definition reads back off
the
stored schema, so a reopened handle needs no side channel. Remote
connections raise NotImplementedError up front rather than failing deep
in
a request, matching the computed-column convention.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
## Summary
- expose `memory_limit` and `num_workers` on the Python FTS
configuration for local builds
- forward both build-only settings to the Lance inverted-index builder
- add an end-to-end regression proving the configured memory budget
reaches the native build
## Root cause
LanceDB 0.26.1 pinned Lance 1.0.1. That Lance version used an FTS
partition-merge path whose retained data made memory grow with merge
progress on very large indexes. Upstream Lance
[#5754](https://github.com/lance-format/lance/pull/5754) changed
partition merging to stream its inputs, reducing peak memory by about
25%. Lance [#6174](https://github.com/lance-format/lance/pull/6174) then
removed the old merge phase, compressed posting lists during
construction, reduced indexing memory by about 60%, and introduced a
total build `memory_limit` for bounded workers.
Current `main` pins Lance 11.0.0-beta.3, which contains those
architectural fixes. This PR does not duplicate or claim the upstream
leak fix; it addresses the remaining Python API gap.
## This repair
LanceDB Python did not expose the native FTS builder resource controls.
`memory_limit` now sets the total local-build budget in MiB, divided
among effective workers, and `num_workers` controls build parallelism.
Both are build-only settings and do not affect remote builds or
persisted index configuration.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo fmt --all`
- `uv run --project python --extra tests --extra dev ruff check .`
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/lancedb/index.py python/python/tests/test_fts.py`
- `uv run --project python --extra tests pytest python/tests/test_fts.py
-q` (51 passed)
Fixes#2923
<!-- lance-gatekeeper-fix:v1 agent=a1ceedf74531e0212cb6f1ebf9390a26
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- document that current LanceDB releases use one compute worker without
warning on two-vCPU containers
- distinguish compute-worker tuning from storage I/O concurrency
- direct users of affected LanceDB 0.21.1 installations to upgrade and
link the current threading guidance
## Root cause
The Lance version bundled with LanceDB 0.21.1 warned whenever the
detected CPU count was less than or equal to its default two-core I/O
reservation. A two-vCPU deployment therefore emitted the warning on
every query even though falling back to one compute worker was the
intended behavior. Lance fixed that warning condition upstream in
lance-format/lance#3710, and LanceDB current main already pins a version
containing the runtime fix; the Python package documentation did not
explain the corrected behavior or the distinct thread controls.
## Validation
- `git diff --check`
- verified the linked Lance threading-model documentation returns HTTP
200
Fixes#2326
<!-- lance-gatekeeper-fix:v1 agent=9f141242416a6dbeb43be0e80404dd4d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`StreamingDataset`, `PermutationBuilder`, and `Permutation` now work
against a `RemoteTable` (LanceDB Cloud and Enterprise), which unblocks
benchmarking the loader against the enterprise cluster cache.
```python
db = lancedb.connect("db://my-db", api_key=..., host_override=...)
ds = StreamingDataset(db.open_table("training"), world_size=8, rank=r)
```
Rows are addressed by `_rowid` exactly as before —
`PermutationReader::load_batch` already built the same `_rowid IN (...)`
filter that `Table::take_row_ids` sends, so the loader's fetch was
always the take path. It just was never allowed to run.
### The guard
`PermutationBuilder.__init__` rejected anything without `_inner`, so a
`RemoteTable` raised `TypeError` before reaching the PyO3 layer — which
already unwraps one via `_table._inner`.
### A bounded schema lookup
`PermutationReader::output_schema` reads the schema off a query plan,
and building a plan on a remote table *executes* the query
(`create_plan` → `execute_query`). With no limit that is `k =
isize::MAX`, so asking a remote table for its output schema pulled the
whole table over HTTP and threw it away — once per assigned split, on
every epoch, since `StreamingDataset.__iter__` constructs a
`Permutation` per split.
One row rather than zero, deliberately: lance gates its limit node on
`self.limit.unwrap_or(0) > 0`, so `Some(0)` means *no limit*.
### Tables with an LSM write spec are refused
A permutation references rows by row id, and rows that have not been
flushed to the base table do not have one yet. The loader could read
around them, but they would then be missing from training with nothing
said about it, so the build refuses such a table up front instead of
half supporting it.
### Fallible identity construction
`PermutationReader::identity` resolved `inner_new` with `unwrap`. That
was near total against a local dataset, but construction counts the base
table — an HTTP round trip for a remote one — so a transient network or
auth failure became a panic across the PyO3 boundary.
### Tests
End-to-end `permutation_builder` and `StreamingDataset` runs against a
mock server, the former torch-free so it runs wherever the suite does,
plus a test that a build succeeds without an LSM write spec and is
refused once one is installed.
A registered `FunctionVersion` has an exact identity and grouped output
contract, but the Python SDK cannot currently bind it to table columns
without manually constructing wire models.
Calling a `FunctionVersion` with named `col(...)` references now returns
one immutable `FunctionApplication` pinned to that exact version. The
application preserves named-struct outputs as one sibling group, while
`rename(columns=...)` defines the result-field to table-column mapping
consumed by `Table.add_columns`. Derived expressions and incomplete or
unknown input names fail before declaration.
Remote Function catalog requests used singular endpoints that are not
exposed by Phalanx. Route registration to `POST /v1/functions/create`
and exact-version lookup to `POST /v1/functions/get`, while preserving
the existing typed Job submission and wait behavior.
## Problem
Generic job result propagation exposed the PyO3 representation of Rust's
unit value as `()` in Python. Unit jobs therefore returned an empty
tuple instead of `None`, breaking the documented `Job.wait()` contract
and the Python doctest workflow.
## Behavior
Unit job completion now converts explicitly to Python `None`. Typed job
results continue to pass through unchanged, with synchronous and
asynchronous regression coverage.
## Problem
The canonical Function wire values and typed remote Job contract do not
yet provide a Python authoring surface or catalog client, so users
cannot package a scalar callable, register it, or reopen the exact
immutable Function version.
## Behavior
This adds scalar-only `@udf` authoring with deterministic annotation or
explicit Arrow schema validation, content-addressed Python artifacts,
and an internal scalar-to-Arrow-batch adapter descriptor. Registration
payloads model non-secret environment values and secret names only.
Remote connections can submit `create_function_async` and receive a
typed `Job<FunctionVersion>`, then reopen that exact version by name and
version ID. Synchronous connections can call `create_function` to submit
and wait for the immutable version in one operation. Local Function
catalog operations return a stable `NotSupported` error. Shared
Rust/Python golden payloads and mocked catalog responses freeze the
request, typed terminal result, and exact lookup contract.
## Validation
- Rust formatting, remote check, clippy, and focused LDB-1/LDB-2 tests
- Python formatting, lint, and focused LDB-1/LDB-2 tests
- Python API documentation build
Function applications from the canonical remote contract cannot
currently declare scalar or grouped computed-column outputs atomically.
This adds the remote-only declaration contract for scalar,
struct-as-one-column, and expanded named-struct outputs. It validates
result mappings, fixes exact input/output Arrow schemas in the request,
persists grouped sibling metadata, and keeps local Function execution
unsupported. Unknown newer application or binding metadata remains
readable, while schema-changing mutations fail closed instead of
rewriting it.
Stable Lance field IDs are deliberately not a declaration prerequisite
in this slice. Inputs bind by parameter name and field path; Sophon
remains responsible for exact-version validation, atomic all-NULL
sibling creation, binding identity and revision allocation, and
persisted output identities.
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2
APIs throughout. This removes dual-version behavior from schema
conversion, query serialization, embedding models, and Function wire
models while preserving their existing public and canonical-wire
behavior.
The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared
compatibility floor remains covered.
PyO3 exposed `LsmWriteSpec` with its default `builtins` module, causing
mkdocstrings to resolve the public `lancedb.LsmWriteSpec` re-export as
`builtins.LsmWriteSpec` and fail the documentation build. Declare the
native extension module and pin the public re-export with a regression
test.
This also applies the repository's current Ruff formatter to seven
previously unformatted Python scripts.
## Problem
Enterprise Function-backed computed columns need a stable SDK contract
before Sophon catalog and execution endpoints can be added. The existing
`Job` API can only represent unit terminal results, and there is no
shared Rust/Python wire definition for immutable Function versions,
applications, bindings, or refresh results.
## Behavior
This introduces remote-only canonical Function values in Rust and
Python, evolves `Job<T = ()>` to decode typed remote terminal results
while keeping local spawned operations unit-typed, and fixes the
cross-language contract with shared JSON golden fixtures. Unknown fields
and discriminator values remain forward-decodable, while canonical
output contains only fields known to the client. Function models contain
secret names only.
Sophon remains the sole owner of catalog persistence, environment bake,
secret resolution, execution, and publication. This PR does not add
authoring/catalog endpoints, local execution, refresh runners, or live
Sophon E2E coverage.
`cargo deny` did not check crate-level dependency declarations against
`[workspace.dependencies]`, so a crate used by both the core crate and
the bindings could be declared independently in each one and drift. For
example `tokio` was pinned at `1.23` in `rust/lancedb` and `1.40` in
`python`, and `pin-project` at `1.0.7` in the workspace table but
`1.1.5` in `python`.
This PR turns on cargo-deny's `bans.workspace-dependencies` lint, which
fails when a dependency is used by more than one member without going
through `workspace = true`, and when a `[workspace.dependencies]` entry
is used by nobody.
Enabling it surfaced 12 violations. Fixing them means adding `bytes`,
`lancedb`, `serde`, `serde_json`, `tempfile`, `tokio`, and `uuid` to
`[workspace.dependencies]`, and pointing the `arrow`, `arrow-buffer`,
`async-trait`, `chrono`, and `pin-project` declarations at the entries
that already existed. `Cargo.lock` is unchanged, so resolution is the
same as before.
The shared `chrono` entry now carries `default-features = false,
features = ["clock"]`, matching what `nodejs` and `python` already asked
for — cargo ignores a member's `default-features = false` unless the
workspace entry sets it too. On the targets we build, `clock` covers
everything `rust/lancedb` was getting from chrono's defaults.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
<!-- lance-gatekeeper-fix:v1 agent=40e5cf476a59265c71653574eda834d2
generation=1 -->
## Summary
- preserve incoming PyArrow `arrow.json` fields while schema
sanitization aligns input to a stored `lance.json` schema
- let Lance perform the required JSONB encoding instead of relabeling
raw JSON bytes as encoded storage
- cover both merge insert and the conditional add sanitization path with
end-to-end regression tests
## Root cause
Python schema sanitization aligns incoming data to the table schema
before passing it to Lance. Merge insert always takes this path, while
add takes it conditionally for preprocessing such as non-default
bad-vector handling or embedding functions. For JSON columns, the cast
changed logical `arrow.json` strings into the table's JSONB-backed
`lance.json` storage type without encoding the bytes, so Lance treated
raw JSON text as JSONB.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-k 'merge_insert or add_sanitization_encodes_json' -q`
- targeted schema-cast and JSON encoding tests
- `ruff check .`
- `ruff format --check python/python/lancedb/table.py
python/python/tests/test_table.py`
Fixes#3923
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Gatefixer
Dan Tasse
Xuanwo
Wyatt Alt
Drew
Lance Release
Jack Ye
Ayush Chaurasia
Weston Pace
Will Jones