Secret CRUD used plural collection routes with the identity in the body, which
is the one shape in the HTTP surface that does not match Table and Function
CRUD. #4166 moved Functions onto `<object>/{id}/<action>`; this does the same
for Secrets, so there is one route grammar rather than one per object type.
POST /v1/secrets/create -> POST /v1/secret/{id}/create
POST /v1/secrets/alter -> POST /v1/secret/{id}/alter
POST /v1/secrets/describe -> POST /v1/secret/{id}/describe
POST /v1/secrets/drop -> POST /v1/secret/{id}/drop
POST /v1/secrets/list -> GET /v1/namespace/{id}/secret/list
`{id}` is the namespace path and the name joined with the client's configured
delimiter, built by the same helper a table identifier is. The name and the
namespace therefore leave every request body: create and alter carry the value
alone, drop and describe carry nothing, and listing moves its `page_token` to a
query parameter. `add_namespace_path` goes with them -- a root Secret is
addressed by its bare name, so there is no longer an empty field to omit.
Identifiers are written into the path as-is rather than percent-encoded. The
charset a Secret name and a namespace segment admit already excludes everything
that would need escaping, and a route that reads like the ones beside it is
worth more than encoding against a charset that does not exist yet. If that
charset widens, this is the place that has to widen with it.
Listing takes a real namespace rather than rejecting a non-root one the way the
Function route does: Secrets are addressable inside nested namespaces, so the
namespace identifier is the whole point of the route.
`build_secret_identifier` is a wrapper over `build_table_identifier` rather than
a second implementation. The grammar belongs to the namespace spec rather than
to tables, and a call site should read as the thing it addresses.
Also removes `_DECLARED_SECRET`, which has been unreferenced since client-side
binding validation was dropped, and whose pattern was the environment variable
grammar rather than a Secret name's.
Tests: lancedb secret unit tests 7, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, 93 Python tests across both slices.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
A Secret is identified by a namespace path plus a name, and resolution is
exact: one under `["prod"]` is not visible from `["prod", "vision"]` and never
falls back to a parent.
The binding names it as a `SecretReference` of `{name, namespace_path}` rather
than one joined string. A joined id needs a delimiter, and a delimiter has to be
excluded from every name and every segment forever, agreed on by both sides, and
re-agreed each time either side grows a new way to be configured --
`ClientConfig.id_delimiter` is exactly that, configurable and advertised on
every request, so an id joined on a fixed `$` would contradict a delimiter the
same request declares. Naming the parts costs one object and settles it.
Every verb takes `namespace_path` keyword-only, defaulting to the root. On
`EnvVarSecret` the same argument is `secret_namespace_path`, matching
`secret_name`: a binding names two things, and the unqualified word reads as
the variable's namespace rather than the Secret's. It pins the path at
construction, so a worker resolves the Secret it was handed rather than
re-resolving against its own default.
The five PyO3 secret methods take `Option<Vec<String>>` with a `None` default,
as every other namespace-taking method on `Connection` already does. They were
the only ones requiring the argument positionally, so the keyword form their
neighbours accept did not work on them.
A root path is omitted from the request body rather than sent empty, so a root
request is byte-identical to one from a client that predates this, and a root
binding to one written before namespaces existed. Tests pin both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
`secret_env_bindings` took the general noun for one delivery mode. A binding is
the concept; how it arrives is a property of one. `secret_bindings` is a list of
`SecretBinding`, tagged by `kind`, so a later mode is a variant rather than a
sibling field -- and the rules that are per-Function, like how many Secrets it
may bind, stay answerable from one place.
The cost of one field is that an unknown kind is a decode error unless it is
caught. It is caught, the way `PythonRuntimeSpec` catches an unknown runtime:
`Unrecognized { kind }`, `#[non_exhaustive]`, and the payload dropped rather
than retained because the client does not proxy catalog values. A test pins it
-- a `file` binding from a newer server decodes, reports its kind, and
round-trips as its discriminator without failing the version around it.
A list has no key order to inherit, and the list is in the version hash, so
`bind_secrets` sorts it: a caller's argument order is not part of what a
Function is.
The Secret is named under `secret_ref`, not `secret`: the service scans Job
payloads for credential-shaped keys and refuses one called `secret` whatever it
holds. That guard is worth more blunt than argued with.
`EnvVarSecret` names it the same way for the same reason: the field is
`secret_name`, not `secret`. In a binding both halves need saying -- which
variable, which Secret -- and the bare noun reads as the credential rather than
the name of one, which is the misreading the type exists to prevent.
`create_secret` and `alter_secret` also stop sharing a request shape. They are
different operations to the service -- one refuses an existing name, the other
requires it -- and either may grow a field the other has no meaning for. What
they share is posting a body that must not be logged, which is a function.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
`SecretInfo` carries `created_at_millis` / `updated_at_millis` as `i64` rather
than RFC 3339 strings, matching `created_at_millis` elsewhere in the platform.
A caller comparing two timestamps no longer parses anything, and the PyO3 layer
hands Python integers instead of stringifying them. `updated_at_millis` is the
only observable that a rotation landed -- no API returns a credential -- so it
is worth being a number a caller can compare.
`RemoteListSecretsRequest` is declared alongside its response instead of being
an inline object built field by field, so a reader of one finds the other.
`function.rs` gains unit tests for the pure parts that had none: canonical JSON
sorts keys at every depth and leaves array order alone -- it is what the version
hash is taken over, so both matter -- floats are refused at any depth, and the
unknown-key check inspects only the level it is handed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
Three things from review.
`SecretInfo` moves to `lancedb::secrets`. It was in `database` because that is
where the trait lives, not because it belongs to a database's shape; a new
object should land in its own module and the Secret verbs will follow it.
Binding and value validation come out of the client. The service owns those
rules -- it owns the runtime the names land in and the store the values go to --
and a copy here could disagree with it without either side noticing. What the
client was checking, the service already rejects: the per-Function cap, the
environment-variable grammar, disjointness from `runtime.env`, and the value
size. The cost is a round trip before the error, and the error is the service's
own words rather than a paraphrase that can drift.
The tests follow the rule rather than the check: what the client guarantees is
that the envelope it sends is the envelope the caller wrote, so that is what is
asserted now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
LanceDB namespace and table names are `[A-Za-z0-9_.-]`, so excluding periods
here put Secrets out of reach inside any namespace a user already has one in --
unreachable to create, alter, describe, drop or bind, with no way to recover
but renaming the namespace.
The rule is that set and nothing further. No positional constraint rides along:
the established grammar says nothing about which character comes first, so a
segment may begin with `_`, `-` or `.` today, and a Secret has to be nameable
wherever a namespace already is. A narrower rule would reintroduce the same
unaddressability it is here to remove, one character class over.
Names and path segments follow one rule, and the client matches the service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
`validate` checked only the binding count, so a caller reaching the low-level
entry point past `bind_secrets` could register an environment variable name the
runtime cannot deliver, or bind a Secret to a name `runtime.env` already sets --
which would resolve by delivery order, with a value visible in the Function's
record and a value that is not.
Secret values are bounded here too, at the limit the service enforces, so an
oversized credential is refused before a request body is built rather than
after it has been serialized and uploaded.
Sandbox-reserved names are deliberately still the service's alone: that list
belongs to the runtime that owns it, and a copy here would drift from it
silently.
Both gate reproducers now fail closed with no request reaching the service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
`log_request` logs any JSON body verbatim at debug, and Python and Node both
wire that logger to `LANCEDB_LOG`. `create_secret` posts the value in its body,
so ordinary SDK debug logging wrote the credential to application logs. The
comment on `write_secret` reasoned correctly about proxy traces and access logs
and missed the logger in this process.
Redaction cannot live in the value model: the logger sees the serialized body,
where the credential is already plaintext bytes. So the request says whether its
body may be logged -- `send_suppressing_body` for the one whose body is the
credential -- and `log_request` obeys rather than deciding. The transport cannot
tell a credential from any other payload, and a list of routes there would have
to be kept in step with endpoints declared elsewhere.
The same debug line prints the request's Debug, which prints headers, so the
API key was in every debug line of every request regardless of route. Marking
the header value sensitive is what stops that.
The end-to-end regression fails without this: the log carried
`,"value":"SECRET_VALUE_SENTINEL"}` verbatim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
`secret_bindings` read as the general record of what a Function needs, but a
`Map<String, String>` keyed on environment-variable name can only ever hold an
env-delivered binding: an accessor binding has no variable to key on. Naming
the field for its delivery mode leaves a later mode a sibling field rather than
a tagged value type, which would break a field already in the identity hash.
The per-Function cap moves from `bind_secrets` to
`Connection::create_function_async`, above the backend dispatch, so it holds
for every language surface rather than only the one that validates first. The
low-level PyO3 entry point reached the wire past the Python check; it no longer
does.
Also corrects a doc comment claiming registration fails when a bound Secret is
absent. Existence is first answered at `add_columns`, by design.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
Adds the client half of database-scoped named Secrets: a Secret is a name and
an opaque value stored by the service, and a Function binds one to the
environment variable its library already reads.
- `db.create_secret` / `alter_secret` / `list_secrets` / `describe_secret` /
`drop_secret` on sync, async and remote connections, with the pyo3 binding
and the Rust client behind them. There is no read API by construction rather
than by policy: no code path returns a stored credential, and
`describe_secret` answers with metadata only.
- `EnvVarSecret(secret=..., env_variable=...)` pairs a Secret with the variable
it arrives in. It is a pure local constructor -- it contacts no server, so it
cannot fail on a Secret that does not exist -- and it exists so a bare string
in that position, which would be a credential, is a TypeError rather than a
plausible-looking mistake that reads identically in a diff.
- `create_function(..., secrets=[...])` carries the bindings as
`secret_bindings`, a map from variable name to Secret name. The value never
travels: it is resolved by the service when the Function runs, which is what
lets a rotation reach columns pinned to an older FunctionVersion.
The UDF body is unchanged and stays portable -- it reads `OPENAI_API_KEY` the
way it always did, and the binding is what puts a value there.
Squashed: the original three commits were a first design plus a rewrite of it,
so their sequence describes an interface that no longer exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XE1UwYKsgbb3USBfkqCE6v
refresh_column fills nulls, so once a row has a value nothing revisits
it: an update to one of its inputs, or a definition change, leaves the
computed value stale for good.
This stamps the column's field metadata with the definition it was
computed under and a per-fragment signature of the input storage it was
read from (input data files and overlays; not the deletion file, since a
delete changes no surviving value). A refresh recomputes every live row
of a fragment whose stamp disagrees with the manifest, then records what
it computed from in a second commit after the fill. A compacted fragment
inherits freshness through the Rewrite lineage when every fragment it
was built from was signed, or was appended since the stamp, never had an
input moved, and left its rows of the product unfilled (a raw append may
supply a value; the product's data is the evidence, and the null fill
covers those rows); otherwise it recomputes. A column declared before
the stamps existed keeps the null-fill contract on its first refresh,
which enrolls it as it stood.
The map is one entry per fragment per column, so it is kept out of the
manifest: each stamp writes an immutable sidecar under `_computed/`,
named by its content digest, and the field metadata holds the digest.
Pruning old versions also drops the sidecars no remaining version
references, keeping any younger than seven days as lance keeps
unverified files, since a sidecar is put before the commit that
references it. The stamp commit is metadata-only, so a materialized
view's drift check treats it like the fill. The core lives in
`table::freshness` so a remote refresh can share the contract.
`lancedb/embeddings/utils.py` documents `max_retries` with "(default is
10)" — the signature default is `7`. Docs-only; conventional title per
the contribution guide.
Co-authored-by: Xuanwo <github@xuanwo.io>
Native job metadata types report `builtins` as their module, so Griffe
cannot resolve the public `lancedb.job` re-exports and the Python API
reference build fails. Set the PyO3 module metadata for `JobInfo`,
`JobDescription`, and `JobFailureInfo`, and cover import resolution in
the existing package metadata tests.
Reproduced the failure and validated the fix with the docs CI toolchain
(`griffe==0.49.0`, `mkdocstrings==0.25.2`, and
`mkdocstrings-python==1.10.9`). After rebuilding the native extension,
the full `PYTHONPATH=. mkdocs build` succeeds and all three classes and
their public members appear in the generated reference.
Align the experimental Function HTTP transport with the equivalent Table
CRUD API shape. This is an intentional breaking change to the
experimental Function routes; public Rust and Python APIs remain
unchanged.
## Route comparison
| Operation | Function before | Function after | Equivalent Table API |
| --- | --- | --- | --- |
| Create | `POST /v1/functions/create` | `POST /v1/function/{id}/create`
| `POST /v1/table/{id}/create` |
| Describe | `POST /v1/functions/describe` | `POST
/v1/function/{id}/describe` | `POST /v1/table/{id}/describe` |
| List | `POST /v1/functions/list` | `GET
/v1/namespace/{id}/function/list` | `GET /v1/namespace/{id}/table/list`
|
| Drop | `POST /v1/functions/drop` | `POST /v1/function/{id}/drop` |
`POST /v1/table/{id}/drop` |
## Contract details
- Create, describe, and drop use a singular resource path. Their `{id}`
path parameter is the URL-encoded Function name, and the duplicate
Function identifier is removed from each request body.
- Create continues to accept `202 Accepted`.
- List changes from a POST with a JSON body to a namespace-scoped GET.
Its `{id}` path parameter is the namespace identifier rather than a
Function name.
- Functions do not support nested namespaces yet, so the client lists
against the root namespace identifier (`$` with the default delimiter).
A non-root namespace is rejected.
- The optional list filter is named `name`. `limit`, `page_token`, and
`include_definition` remain available as query parameters.
- The paginated list response shape is unchanged.
Adds [typos](https://github.com/crate-ci/typos) as a CI check and
pre-commit hook, the same way Lance does it, so misspellings like the
ones fixed in #4146 get caught automatically going forward.
This also fixes the misspellings `typos` found across the repo (Rust,
Python, TypeScript source, comments, and generated docs), and adds a
small `.typos.toml` with `extend-words` entries for terms that are
correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ`
(a real quantization algorithm name), `mmaped` (the actual name of a
`candle-core` API we call), and `Writeable` (from Python's
`_typeshed.WriteableBuffer`). Third-party license files are excluded.
Fixes#4147
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Fixes#4027
## Summary
`AsyncHybridQuery`
(`table.query().nearest_to(...).nearest_to_text(...)`) paginates
incorrectly when `.offset()` is used: the second page repeats rows from
the first page and silently drops others.
`offset()` on a hybrid query pushes the offset down into *both*
sub-queries (`HybridQuery::offset` in `python/src/query.rs` forwards to
`inner_vec` and `inner_fts`), so each sub-query independently skips its
own first `offset` rows before the results are fused.
`AsyncHybridQuery.to_batches` then called `_combine_hybrid_results(...,
limit=self._inner.get_limit())` without an `offset`, so the reranked
table was sliced starting at position 0 and the sub-query limits were
never raised to cover the skipped prefix.
On the 4-row fixture in `test_hybrid_query.py`, with `_rowid` ordering
`[3, 0, 2, 1]`:
| query | before | after |
| --- | --- | --- |
| `.limit(2)` | `[0, 3]` | `[0, 3]` |
| `.offset(2).limit(2)` | `[3, 1]` | `[2, 1]` |
Row `3` was returned on both pages and row `2` was never returned at
all.
This is the async counterpart of #3769 (`Fixes #3765`), which fixed the
same bug in the synchronous `LanceHybridQueryBuilder`. #3765 explicitly
deferred the async path; this PR closes that gap and reuses the `offset`
parameter that #3769 already added to `_combine_hybrid_results`. The
synchronous path is unaffected — it was fixed in #3769.
## Changes
`python/python/lancedb/query.py`, `AsyncHybridQuery.to_batches`:
- Each sub-query now fetches `limit + offset` rows and its own offset is
reset to 0, so the fused result contains the full prefix the window is
sliced out of.
- The combined, reranked table is sliced with `offset=` instead of
always starting at 0.
Both halves are needed: raising the sub-query limits without the final
slice still returns page 1, and slicing without raising the limits still
misses rows.
`nodejs` has no equivalent hybrid combine path, so there is no SDK
parity gap here.
## Test plan
- [x] New regression test `test_async_hybrid_query_offset` in
`python/python/tests/test_hybrid_query.py`, mirroring the sync
`test_hybrid_query_offset`. It asserts the offset window is a suffix of
the un-offset result *and* that page 1 + page 2 together cover every row
exactly once (a row-count-only assertion would pass even with
duplicates).
- [x] `pytest python/tests/test_hybrid_query.py` — 16 passed
- [x] `pytest python/tests/test_rerankers.py` — 9 passed, 11 skipped
- [x] `pytest python/tests/test_query.py` — 86 passed
- [x] `pytest --doctest-modules python/lancedb/query.py` — 13 passed
- [x] `ruff format --check` / `ruff check` — clean
---
## Scope, after review
@lancedb-gatekeeper raised three points. Two were mine and are fixed in
`04d07c2`; the third is deliberately left alone and I'd like a
maintainer's call on it.
**Fixed — effective limit was read from the FTS child only.**
`HybridQuery::get_limit()` (`python/src/query.rs:1159`) returns
`self.inner_fts.inner.current_request().limit`, so an FTS-first hybrid
with no explicit `.limit()` yielded `None`, skipped the widening branch
and passed `limit=None` to the combiner — returning the union of both
candidate lists instead of the documented default of 10. The limit is
now derived from both children with a `DEFAULT_HYBRID_LIMIT = 10`
fallback, so construction order no longer matters.
**Fixed — `explain_plan()` / `analyze_plan()` described a different
query than the one that ran.** Both built their children straight from
`self._inner`, bypassing the limit/offset rewrite in `to_batches`, and
reported `skip=2, fetch=2` while execution used `skip=0, fetch=4`. Child
preparation now lives in one `_create_child_queries()` helper used by
all three.
> **Visible change to `explain_plan()` output:** because the plan is now
built from the real execution children, which carry `with_row_id()`, the
two `ProjectionExec` lines gain a `_rowid` column. The doctest is
updated to match. This is the diagnostic becoming truthful rather than
the assertion being weakened — it is still an exact-match comparison.
**Not fixed here — RRF candidate-pool invariance.** Widening each
sub-query to `limit + offset` does change the candidate pool between
page requests, so the fused ranking can shift and pagination can still
repeat rows. That's a real problem, but it is exactly what the merged
sync path does today:
```python
# LanceHybridQueryBuilder (sync), merged in #3769
sub_query_limit = self._limit + (self._offset or 0)
```
Making the pool invariant means choosing a contract — a fixed candidate
pool, or an explicit cursor — and that ought to apply to sync and async
together rather than letting the two paths diverge. I've asked in the
review thread which way you'd prefer, and I'm happy to do it here or in
a follow-up covering both paths.
So, to be precise about what this PR delivers: it makes `.offset()` take
effect on the async hybrid path and makes the diagnostics honest. It
does not make hybrid pagination stable across pages under reranking —
that needs the contract decision above.
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core from v12.0.0-beta.14 to
[v12.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.15).
No compatibility fixes were required; `cargo clippy --quiet --workspace
--tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and
`git diff --check` passed.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.
## Before
Four ways to ask about a job, none of which told you much.
```python
job = table.refresh_column_async("embedding")
job.status() # "running". That was the entire debug surface.
db.get_job(job_id) # state, and a spec. No result, no progress.
db.job_history(job_id) # raw record batches, no limit, no filter
db.job(job_id) # a handle that knew nothing
```
## After
Open a job the way you open a table; the handle answers everything.
```python
job = db.open_job(job_id) # raises JobNotFoundError if there is no such job
```
```python
>>> print(job)
Job(
id='job-1',
state='failed',
job_type='refresh_column',
creation_ms=1757000000000,
spec={
"column": "embedding",
"num_workers": 4
},
failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```
Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.
Per-fragment progress *while it is still running*:
```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist() # [5000, 5000, ...]
```
The handle an async action returns is the same object, one `refresh()`
away:
```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```
TypeScript is the same experience, down to `console.log`:
```ts
const job = await db.openJob(jobId); // rejects if there is no such job
console.log(job); // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```
## Why each piece matters
- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.
`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.
## Breaking
All shipped in 0.38.0. No deprecated aliases.
| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |
Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
This PR stops blob table create from implicitly enabling stable row ids.
A blob schema still selects Lance file format 2.2, but row id behavior
stays with the table config.
Compact then fetch with a `_rowid` captured before compaction is still
not supported on a default table. That needs `take` to remap row
addresses through blob reuse rather than making stable row ids a
blob-table default.
BREAKING CHANGE: blob create no longer enables stable row ids. A blob
schemastill selects Lance file format 2.2. Fetch uses `_rowid` on HEAD.
Held ids survive compact only when the table has stable row ids.
## Testing
* `cargo fmt --all`
* `ruff format .`
* `ruff check .`
* `cargo clippy --quiet --features remote --tests --examples -p lancedb`
* `cargo test --quiet --features remote -p lancedb --test
blob_integration`
* `python/.venv/bin/pytest python/python/tests/test_blob.py -q`
Supports fully nullable named Function outputs while preserving the
distinction between a valid all-null struct and a null/unassigned
result.
## Concrete example
This UDF contract is now valid:
```python
@udf(
input_schema=pa.schema([
pa.field("text", pa.string(), nullable=False),
]),
output_schema=pa.schema([
pa.field(
"embedding",
pa.list_(pa.float32(), list_size=1024),
nullable=True,
),
pa.field("embedding_failure_reason", pa.string(), nullable=True),
pa.field("embedding_failure_code", pa.int32(), nullable=True),
]),
)
def embed(text):
...
```
A successful row can return:
```text
embedding = [0.12, ...]
embedding_failure_reason = NULL
embedding_failure_code = NULL
```
If remote inference still fails after retries, it can return:
```text
embedding = NULL
embedding_failure_reason = "HTTP 429: rate limited"
embedding_failure_code = 429
```
An all-null but valid result struct is also assigned; it is not mistaken
for unfinished work.
## Binding shapes
- Mapping the result to one output column stores the `StructArray`
directly, including its parent validity bitmap.
- Flattening the result into top-level columns stores the parent
validity in a reserved internal nullable Boolean assignment column that
is not part of the UDF result mapping.
- An outer null struct remains unassigned/skipped. A valid struct
remains assigned regardless of which child fields are null.
- Scalar Function outputs remain non-nullable.
The contract is preserved through Python registration, Rust application
planning, persisted `FunctionBinding` metadata, schema revalidation, and
Enterprise execution.
## Summary
Add SQL execution to remote LanceDB connections. On the standard
synchronous connection, `execute_query` waits for the initial result
stream and returns its Arrow reader. `execute_query_async` is called
without Python `await` and immediately returns a query handle for status
inspection, streaming, or cancellation. Local databases report that SQL
is not supported.
The transport and query lifecycle live in Rust. Python exposes
native-backed synchronous and asynchronous connection methods and query
wrappers; it does not use PyArrow's Flight client.
## User experience
The standard synchronous connection supports both direct reads and
background query execution:
```python
db = lancedb.connect(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
# Direct execution waits only until the initial result stream is available.
# Later batches continue streaming as the query progresses.
reader = db.execute_query(
"SELECT * FROM events",
default_namespace_path=["production"],
)
for batch in reader:
print(batch.num_rows)
# Background execution returns a query handle immediately. Despite the
# `_async` suffix, no Python `await` is needed on a synchronous connection.
query = db.execute_query_async("SELECT * FROM events")
print(query.id)
description = db.describe_query(query.id)
print(description.status)
print(description.progress)
print(description.expires_at)
# Start reading as soon as the service advertises partial results. The reader
# continues polling and yields newly available record batches until the query
# and all result endpoints are complete.
reader = query.reader()
for batch in reader:
print(batch.num_rows)
# Or cancel a different still-running query. Its status becomes "cancelling"
# while the server is still working, then "cancelled" once confirmed.
cancelled_query = db.execute_query_async("SELECT * FROM large_events")
cancelled_query.cancel()
```
The less commonly used asynchronous connection exposes the same
operations as coroutines:
```python
async_db = await lancedb.connect_async(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
query = await async_db.execute_query_async("SELECT * FROM events")
async for batch in await query.reader():
print(batch.num_rows)
```
The UUIDv7 query id is scoped to the connection that submitted it. The
connection retains lightweight shared query state used by
`query.describe()` and `db.describe_query(query.id)`; the id does not
encode SQL or a Flight continuation token and is not a cross-connection
resume token. Abandoned state has bounded retention, and terminal state
remains available briefly.
Unqualified table names use the connected database and the `public`
namespace by default. `default_namespace_path` accepts a list such as
`["production", "events"]`. SQL can still use qualified names to
reference other databases and namespaces available to the deployment.
## Design
- Uses Arrow Flight `PollFlightInfo` for submission and long polling,
`DoGet` for results, and `CancelFlightInfo` for cancellation. Each
`PollInfo.info` is treated as the cumulative set of currently available
endpoints, so advertised tickets are consumed once and batches can be
delivered before execution is complete.
- Serializes result completion and cancellation into one lifecycle. A
server-accepted request reports `cancelling` and wakes blocked
status/result work; a later retry can confirm `cancelled`. Result
retrieval is rejected after cancellation is accepted, while cancellation
after a result was already delivered is a no-op.
- Assigns a time-ordered UUIDv7 connection-scoped query id and retains
only shared evolving lifecycle state, keeping SQL, Flight continuation
tokens, and Arrow result data out of public ids and the registry.
- Leaves admission control to the server while honoring server
expiration and a local fallback retention window for abandoned entries.
- Retains terminal ids for five minutes so they remain available for
connection-level description.
- Keeps one lazily initialized SQL client on each remote database
connection and attaches fresh authentication, routing, namespace, and
request metadata to every operation.
- Applies the configured overall timeout to each execution, description,
reader, and cancellation operation. A result reader carries one absolute
deadline from `reader()` through the end of streaming; connect and read
timeouts continue to bound their individual phases.
- Returns a bounded, backpressured, single-consumer Arrow stream rather
than collecting the full result in memory. Dropping the reader stops
downloading but does not implicitly cancel the server query.
- Preserves typed schemas for empty result sets through the stream
schema.
- Accepts Flight result messages up to 1 GiB so a valid row containing a
large blob, string, or vector is not rejected by tonic's 4 MiB default
receive limit.
- Supports the Python client first while keeping the authoritative
implementation in the Rust core.
Merge insert has always taken a list of columns to match on, and local
tables have always joined on all of them. Remote tables did not: any
list longer than one was rejected with `MergeInsertBuilder only supports
a single 'on' column`, so a composite-key upsert was impossible against
LanceDB Cloud and Enterprise from Rust, Python or TypeScript.
The remote request now carries `on` as a list and sends it as one
repeated query parameter per column — `?on=shard_key&on=id`. That is how
the lance-namespace spec encodes an array-valued `on`, so the server
receives a composite key in the shape it expects. A single column still
serializes to `?on=id`, exactly what clients sent before, so existing
callers are unaffected. A column repeated within `on` is now rejected
client-side rather than sent for the server to reject with a 400.
No binding changes were needed: `Table.merge_insert` in Python and
`Table.mergeInsert` in TypeScript already accepted a list, it just could
not reach a remote table. Both gain a test for composite keys, and the
doc comments now say what passing several columns means.
Part of
[ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the).
## Example
```python
table.merge_insert(["shard_key", "id"]) \
.when_matched_update_all() \
.when_not_matched_insert_all() \
.execute(new_data)
```
A row whose `id` matches an existing row but whose `shard_key` differs
is an insert, not an update.
## Not included
Java. Java callers reach merge insert through
`org.lance.namespace.LanceNamespace`, whose
`MergeInsertIntoTableRequest.on` is a single string until
lance-namespace 0.12
([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363),
[lance#8915](https://github.com/lance-format/lance/pull/8915)). There is
nothing in this repo's Java SDK to change until the `lance-core` pin can
move.
Sending more than one column requires a server that accepts the repeated
parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571));
an older server returns a 400 rather than silently merging on one
column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remote create-index requests already expose `replace` on the builder,
but the remote client did not consistently forward an explicit
`replace=false` over REST. That meant create-only intent could be lost
before it reached a remote server, even though local builders and Python
APIs can express it. This PR forwards `replace=false` on the existing
`create_index` endpoint and keeps the current default behavior unchanged
for compatibility.
This was accomplished with the following changes:
- Serialize `replace: false` into the existing remote create-index
request body when the builder is configured with `.replace(false)`.
- Forward `replace` through the synchronous Python remote `create_index`
wrapper so `RemoteTable.create_index(..., replace=False)` reaches the
repaired path.
- Continue omitting `replace` for the default path so existing remote
create-index requests keep their current semantics.
- Document `name` and `replace` on the existing OpenAPI create-index
request schema.
- Add coverage that verifies the remote client uses the existing
`/create_index/` route and forwards `replace=false`, including the
synchronous Python unified API.
### Testing
- `cargo fmt --all --check`
- `cargo test -p lancedb --features remote
test_create_index_forwards_replace_false_on_existing_route --locked`
- `uv tool run maturin develop --extras tests,dev,embeddings`
- `uv run --frozen pytest
python/tests/test_remote_db.py::test_remote_create_index_new_api`
- `uv run ruff format --check python/lancedb/remote/table.py
python/tests/test_remote_db.py`
- `cargo build -p lancedb --features remote --locked`
- `cargo clippy -p lancedb --features remote --all-targets --locked --
-D warnings`
Function signatures currently reject Blob v2 fields nested inside
structs, preventing UDFs from accepting or returning structured values
that contain blobs.
Accept canonical Blob v2 fields as direct or recursive struct children
while preserving exact field metadata and nullability. Blob fields under
list, large-list, fixed-size-list, or map ancestors remain rejected
because collection runtime adaptation is outside the supported Function
ABI.
A whole named struct result can bind directly to one destination column
without introducing an extra wrapper level.
Function registration and exact lookup are exposed through the SDK, but
clients cannot discover published versions even though the server
provides `POST /v1/functions/list`.
Add Rust and Python sync/async `list_functions()` APIs that return typed
`FunctionVersion` values. The remote client requests canonical
definitions and follows opaque page tokens until the listing is
complete, including empty intermediate pages, while preserving the
server's name/version ordering. Local databases retain the existing
Function-catalog unsupported error.
The SDK consumes protocol pagination internally so callers receive the
complete catalog rather than handling server-specific page tokens.
A view definition recorded its source by bare name and refresh resolved
that name at the root, so declaring a view over a namespaced source was
refused outright -- materialized views were root-only for every caller.
The definition now carries `source_namespace`, and refresh opens the
source at that coordinate. `plan` takes the namespace too: refresh
re-plans the stored definition and persists the result when it migrates,
so defaulting it there would strand the view on its next rebuild.
The stored kind is the version boundary. Root definitions keep the
`select` form byte-for-byte, so everything written before this change
reads exactly as it always did. A namespaced source is stored as
`namespaced_select`: released readers drop unknown fields and resolve a
`select` source at the root, so keeping the old kind would let a
rolled-back worker refresh a view from a same-name root table -- the new
kind routes them to their existing unrecognized-kind refusal instead.
The Python and Node definition parsers learn the new kind alongside the
Rust core.
## Summary
- preserve repeated table offsets without adding a public ordering
guarantee
- retain exact requested ordering in identity and persisted permutations
- cover local, projected, multi-batch, and mocked-remote query paths
## Root cause
Take queries lowered offsets to a set-like IN predicate and discarded
repeated occurrences. Persisted permutation loading also compared the
distinct base-table result count with the requested occurrence count,
rejecting repeated row IDs before its existing reordering step could
expand them.
## Fix
The shared take-query path now deduplicates the predicate for efficient
lookup, requests row-offset metadata internally, and expands each
matching row to the requested multiplicity in backend result order. An
internal opt-in keeps exact requested order for identity
PermutationReader reads, while persisted permutations continue using
their existing ordering map.
## Validation
- cargo test --quiet --features remote --tests
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
- targeted Python local and mocked-remote regression tests
- exact issue reproduction
Fixes#2820
<!-- lance-gatekeeper-fix:v1 agent=75acf840afa6f4be4bff98b567b504bd
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Other changes
### What changed?
- Add a subprocess regression harness for an ordinary `@udf` function
defined in `__main__`.
- Verify the full registration request, artifact digest, and Function
signature stay identical across independent Python processes and
renamed/moved script paths.
- Verify body, referenced-global, and annotation changes still produce
distinct artifact identities, with annotation changes also producing a
distinct Function signature.
### Why is the change needed?
[ENT-2441](https://linear.app/lancedb/issue/ENT-2441/make-sure-function-defined-in-main-gets-stable-signature)
tracks the stability guarantee. Investigation on the exact `840e1d73`
main base found that LanceDB already packages canonical source instead
of cloudpickle bytes, so the unchanged `__main__` function is stable and
no production-code fix is needed. This change closes the missing
regression-test coverage.
[GEN-950](https://linear.app/lancedb/issue/GEN-950/class-based-udfs-defined-in-main-get-a-new-auto-version-on-every-run)
remains a separate Geneva checkpoint-version issue for class-based
callables. LanceDB's Function API continues to accept synchronous Python
functions only.
## Validation
- `cd python && uv run --extra tests pytest
python/tests/test_first_class_function_slice2.py -q` (`40 passed`)
- `uv run --project python --extra dev ruff format .`
- `uv run --project python --extra dev ruff check .` (`All checks
passed!`)
Teach the Python Function signature emitter to serialize PyArrow large
strings as the canonical Arrow type name `large_utf8`.
Extend the shared Function Arrow type fixture and explicit-schema
coverage for scalar, nested, list, and large-list compositions.
Make Function authoring and declaration planning treat Blob v2 as a
scalar semantic type while preserving exact Blob metadata in binding
schemas.
Covers scalar Blob outputs, expanded named-struct outputs, and
whole-result structs with Blob children.
Functions can describe their Python environment today, but cannot
declare accelerator requirements. That prevents Sophon from scheduling
computed-column UDF refreshes onto GPU workers from the immutable
Function definition.
Add `num_gpus` to Python `@udf` through a typed
`FunctionResourceRequirements` value and represent resource-aware
definitions with the `python_v2` runtime discriminator. CPU Functions
retain their existing `python` encoding and canonical identity.
The new discriminator is intentional for mixed-version safety:
deployments that do not understand execution resources reject the
runtime instead of accepting a new field and silently running the
Function on CPU. Required resources are part of Function version
identity; priority, concurrency, and retry policy remain Job concerns.
The actual resource scheduling remains owned by Sophon.
Teach Python Function authoring to retain the compact V1 grammar for
existing types and emit canonical exact JSON for nested struct
signatures. Adds coverage for recursive struct/list schemas and exact
field properties.
Computed-column planning currently sees Blob v2 storage descriptors, so
expressions cannot consume payload bytes or preserve Blob semantics in
their outputs.
A computed declaration now derives its output field from its expression.
A direct projection of a Blob v2 field inherits the source field's Blob
metadata; other expressions retain their ordinary Arrow-inferred type.
Declarations remain ordered, so the same rule applies across chained
projections.
Refresh materializes referenced Blob inputs as `LargeBinary` payload
bytes and publishes inherited Blob outputs through Lance's Blob
conversion path. Remote requests remain within the shared namespace
contract as `{name, computed}`; the server planner is being updated in
tandem to implement the same Blob-aware planning semantics, and remote
enablement must be aligned with that server rollout.
The existing null-as-unfilled contract remains unchanged. Row-level
freshness and cell flags remain follow-up work.
Jonathan M HsiehandClaude Opus 5
Lance Release
Wyatt Alt
陈志谦andXuanwo
Jack Ye
Will JonesandClaude Sonnet 5
Madan Kumar
LanceDB RobotandJack Ye
Drew
Bruno Ramirez
lancedb-gatefixer[bot]andXuanwo
Yang Cen