Compare commits

...

13 Commits

Author SHA1 Message Date
Yang Cen a5fc5b544f Merge commit '61e6614c0916e11353e3cbf1c7c2d1045934d201' into yang/udf-secret-sophon-integration 2026-08-28 11:03:28 +08:00
Jack Ye 61e6614c09 chore: retain computed column support in beta.4 update 2026-08-27 17:24:32 -07:00
Yang Cen f6bfe42ec4 Merge branch 'yang/udf-secret-api' into yang/udf-secret-sophon-integration 2026-08-28 08:16:41 +08:00
Yang Cen ef752c2e3f Merge remote-tracking branch 'origin/main' into yang/udf-secret-api 2026-08-28 08:13:12 +08:00
Yang Cen 15abff3718 Merge commit 'a10c2e39b98981824db3519bbb2381e930720332' into yang/udf-secret-sophon-integration 2026-08-28 07:51:48 +08:00
Yang Cen 0f6a355593 Merge remote-tracking branch 'origin/main' into yang/udf-secret-api 2026-08-28 07:46:10 +08:00
lancedb automation d41ed98b52 chore: update lance dependency to v12.0.0-beta.4 2026-08-27 21:26:01 +00:00
Wyatt Alt a10c2e39b9 feat: let a computed-column batch read its own earlier declarations
`add_columns().computed()` accepted several columns in one call but bound
each against the table's schema as it stood before the call, so `a` and
`b = a + 1` had to be two commits. A server staging declarations behind
other schema work has no atomic way to do that, and a caller reading the
builder's plural signature reasonably expects the batch to be one.

Each accepted column now joins the schema the next one resolves against,
so the batch is planned and committed as one. Order is the dependency
order; reading ahead is still an unknown column. `validate_declarations`
exposes the schema-level checks -- the Function-binding guard and the
planning -- without a commit, for callers that must reject before earlier
work in the same request lands; LSM state is table state and stays a
commit-time check.

Dependent columns need refresh to be dependency-aware, or `b =
coalesce(a, 0)` refreshed before `a` bakes zeros from `a`'s placeholder
null and the fill-once contract keeps them. Refresh now walks the
dependency graph once and fills each reachable column once in dependency
order, each as its own commit, then the requested column. Every fill in
the pass covers only the fragments of the snapshot the pass started from:
a commit may rebase over a concurrent append, and the fragment that admits
carries placeholder nulls no earlier fill covered, so it waits for a later
refresh and is reported as `rows_remaining`. Two concurrent fills of one
input collide on its field in lance's conflict check, so a dependent fill
can only commit over inputs that were durable when it read them.
`rows_filled` counts the requested column only; the async result's
`published_version` is the last commit of the pass, inputs included.
2026-08-27 20:46:32 +00:00
Yang Cen a6ec35502a fix(functions): complete secret validation parity 2026-08-28 00:09:04 +08:00
Yang Cen a0bb1f7597 fix(functions): harden Rust secret submissions 2026-08-27 23:03:27 +08:00
Yang Cen 2562e117b2 fix(functions): harden UDF secret submissions 2026-08-27 21:52:31 +08:00
Yang Cen 134a265ee2 feat(functions): support UDF secret values 2026-08-27 20:58:53 +08:00
lancedb automation c05da95d4c chore: update lance dependency to v12.0.0-beta.2 2026-08-26 01:31:44 +00:00
20 changed files with 1432 additions and 138 deletions
Generated
+42 -42
View File
@@ -3455,8 +3455,8 @@ checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "fsst"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"rand 0.9.5",
@@ -4815,8 +4815,8 @@ checksum = "e037a2e1d8d5fdbd49b16a4ea09d5d6401c1f29eca5ff29d03d3824dba16256a"
[[package]]
name = "lance"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arc-swap",
"arrow",
@@ -4888,8 +4888,8 @@ dependencies = [
[[package]]
name = "lance-arrow"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -4911,7 +4911,7 @@ dependencies = [
[[package]]
name = "lance-arrow-scalar"
version = "58.0.0"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -4925,7 +4925,7 @@ dependencies = [
[[package]]
name = "lance-arrow-stats"
version = "58.0.0"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"arrow-schema",
@@ -4934,8 +4934,8 @@ dependencies = [
[[package]]
name = "lance-bitpacking"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrayref",
"crunchy",
@@ -4945,8 +4945,8 @@ dependencies = [
[[package]]
name = "lance-core"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -4983,8 +4983,8 @@ dependencies = [
[[package]]
name = "lance-datafusion"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow",
"arrow-array",
@@ -5013,8 +5013,8 @@ dependencies = [
[[package]]
name = "lance-datagen"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow",
"arrow-array",
@@ -5031,8 +5031,8 @@ dependencies = [
[[package]]
name = "lance-derive"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"proc-macro2",
"quote",
@@ -5041,8 +5041,8 @@ dependencies = [
[[package]]
name = "lance-encoding"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-arith",
"arrow-array",
@@ -5075,8 +5075,8 @@ dependencies = [
[[package]]
name = "lance-file"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-arith",
"arrow-array",
@@ -5107,8 +5107,8 @@ dependencies = [
[[package]]
name = "lance-index"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arc-swap",
"arrow",
@@ -5172,8 +5172,8 @@ dependencies = [
[[package]]
name = "lance-index-core"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"arrow-schema",
@@ -5195,8 +5195,8 @@ dependencies = [
[[package]]
name = "lance-io"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow",
"arrow-array",
@@ -5236,8 +5236,8 @@ dependencies = [
[[package]]
name = "lance-linalg"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"arrow-schema",
@@ -5251,8 +5251,8 @@ dependencies = [
[[package]]
name = "lance-namespace"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow",
"async-trait",
@@ -5264,8 +5264,8 @@ dependencies = [
[[package]]
name = "lance-namespace-impls"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow",
"arrow-ipc",
@@ -5318,8 +5318,8 @@ dependencies = [
[[package]]
name = "lance-select"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -5333,8 +5333,8 @@ dependencies = [
[[package]]
name = "lance-table"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow",
"arrow-array",
@@ -5374,8 +5374,8 @@ dependencies = [
[[package]]
name = "lance-testing"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"arrow-array",
"arrow-schema",
@@ -5388,8 +5388,8 @@ dependencies = [
[[package]]
name = "lance-tokenizer"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.4#474ae88dd75c437d5d506859b627510978e502e9"
dependencies = [
"frostem",
"icu_segmenter",
+14 -14
View File
@@ -13,20 +13,20 @@ categories = ["database-implementations"]
rust-version = "1.91.0"
[workspace.dependencies]
lance = { "version" = "=12.0.0-beta.2", default-features = false, "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-core = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-datagen = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-file = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-io = { "version" = "=12.0.0-beta.2", default-features = false, "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-index = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-linalg = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace-impls = { "version" = "=12.0.0-beta.2", default-features = false, "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-table = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-testing = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-datafusion = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-encoding = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-arrow = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance = { "version" = "=12.0.0-beta.4", default-features = false, "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-core = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-datagen = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-file = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-io = { "version" = "=12.0.0-beta.4", default-features = false, "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-index = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-linalg = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace-impls = { "version" = "=12.0.0-beta.4", default-features = false, "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-table = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-testing = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-datafusion = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-encoding = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-arrow = { "version" = "=12.0.0-beta.4", "tag" = "v12.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lancedb = { path = "rust/lancedb", default-features = false }
ahash = "0.8"
# Note that this one does not include pyarrow
+1 -1
View File
@@ -28,7 +28,7 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<arrow.version>15.0.0</arrow.version>
<lance-core.version>12.0.0-beta.2</lance-core.version>
<lance-core.version>12.0.0-beta.4</lance-core.version>
<spotless.skip>false</spotless.skip>
<spotless.version>2.30.0</spotless.version>
<spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
+38 -7
View File
@@ -16,6 +16,7 @@ from typing import (
Iterable,
List,
Literal,
Mapping,
Optional,
Union,
)
@@ -687,17 +688,35 @@ class DBConnection(EnforceOverrides):
"""
raise NotImplementedError("serialize is not supported for this connection type")
def create_function(self, definition: UdfDefinition) -> FunctionVersion:
def create_function(
self,
definition: UdfDefinition,
*,
secrets: Optional[Mapping[str, str]] = None,
) -> FunctionVersion:
"""Register a scalar Python UDF and wait for its immutable version.
``secrets`` must contain exactly the names declared by
``@udf(secrets=[...])``. Values are sent in the create request and
stored server-side in the private execution artifact; returned
Function and Job metadata contain only the declared names.
This is the blocking counterpart of :meth:`create_function_async`.
Local connections raise ``NotImplementedError``.
"""
return self.create_function_async(definition).wait()
return self.create_function_async(definition, secrets=secrets).wait()
def create_function_async(self, definition: UdfDefinition) -> Job[FunctionVersion]:
def create_function_async(
self,
definition: UdfDefinition,
*,
secrets: Optional[Mapping[str, str]] = None,
) -> Job[FunctionVersion]:
"""Register a scalar Python UDF through the remote Function catalog.
``secrets`` must contain exactly the names declared by
``@udf(secrets=[...])``. Values are sent in the create request and
stored server-side in the private execution artifact; returned
Function and Job metadata contain only the declared names.
Submission returns a typed job. The immutable Function version becomes
available only when :meth:`Job.wait` succeeds. Local connections raise
``NotImplementedError``.
@@ -1405,8 +1424,13 @@ class LanceDBConnection(DBConnection):
return Job(self._conn.job(job_id))
@override
def create_function_async(self, definition: UdfDefinition) -> Job[FunctionVersion]:
job = LOOP.run(self._conn.create_function_async(definition))
def create_function_async(
self,
definition: UdfDefinition,
*,
secrets: Optional[Mapping[str, str]] = None,
) -> Job[FunctionVersion]:
job = LOOP.run(self._conn.create_function_async(definition, secrets=secrets))
return Job(job)
@override
@@ -2225,17 +2249,24 @@ class AsyncConnection(object):
return AsyncJob(self._inner.job(job_id))
async def create_function_async(
self, definition: UdfDefinition
self,
definition: UdfDefinition,
*,
secrets: Optional[Mapping[str, str]] = None,
) -> AsyncJob[FunctionVersion]:
"""Register a scalar Python UDF through the remote Function catalog.
``secrets`` must contain exactly the names declared by
``@udf(secrets=[...])``. Values are sent in the create request and
stored server-side in the private execution artifact; returned
Function and Job metadata contain only the declared names.
The returned typed job resolves to the immutable Function version.
Local connections raise ``NotImplementedError``.
"""
if not isinstance(definition, UdfDefinition):
raise TypeError("create_function_async requires a @udf definition")
inner = await self._inner.create_function_async(
definition.registration_request.to_canonical_json()
definition._submission_json(secrets)
)
return _typed_job(inner, FunctionVersion.from_json)
+106 -6
View File
@@ -4,7 +4,7 @@
"""Canonical Function values exchanged with LanceDB Enterprise services.
These immutable models contain client/wire state only. Catalog persistence,
environment bake, and execution are owned by Sophon.
environment bake, secret resolution, and execution are owned by Sophon.
``RefreshColumnResult`` is also the backend-neutral result of a local
expression-backed refresh job.
"""
@@ -229,7 +229,7 @@ class PythonEnvironmentSpec(_RemoteValue):
class PythonRuntimeSpec(_RemoteValue):
"""Remote runtime definition with environment values.
"""Remote runtime definition with non-secret environment values.
V1 supports ``kind="python"``. Newer runtime kinds remain readable, while
their unknown payload fields are intentionally not retained by the client.
@@ -268,6 +268,7 @@ class FunctionVersion(_RemoteValue):
runtime: PythonRuntimeSpec
runtime_digest: str
environment_digest: str
required_secrets: tuple[str, ...] = ()
created_at: str
def __call__(self, **inputs: Any) -> FunctionApplication:
@@ -329,12 +330,17 @@ class FunctionVersion(_RemoteValue):
class FunctionRegistrationRequest(_RemoteValue):
"""Stable remote registration envelope produced by :func:`udf`."""
"""Stable remote registration envelope produced by :func:`udf`.
Only secret names are represented. Secret values are supplied separately
when the definition is submitted and are not part of this durable value.
"""
name: str
artifact: FunctionArtifactRequest
signature: FunctionSignature
runtime: PythonRuntimeSpec
required_secrets: tuple[str, ...] = ()
class FunctionVersionRef(_OpenRemoteValue):
@@ -479,6 +485,27 @@ class RefreshColumnResult(_RemoteValue):
_FUNCTION_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_.-]*$")
_SECRET_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
# Keep this byte limit aligned with Sophon's MAX_FUNCTION_SECRET_VALUE_BYTES.
_MAX_FUNCTION_SECRET_VALUE_BYTES = 64 * 1024
_MAX_FUNCTION_SECRET_VALUES_BYTES = 512 * 1024
def _validate_secret_value(name: str, value: Any) -> str:
"""Validate one secret value before building the create request."""
if not isinstance(value, str):
raise TypeError(f"Function secret {name!r} value must be a string")
if not value:
raise ValueError(f"Function secret {name!r} value must be non-empty")
if "\0" in value:
raise ValueError(f"Function secret {name!r} value must not contain NUL")
value_bytes = len(value.encode("utf-8"))
if value_bytes > _MAX_FUNCTION_SECRET_VALUE_BYTES:
raise ValueError(
f"Function secret {name!r} value exceeds the "
f"{_MAX_FUNCTION_SECRET_VALUE_BYTES}-byte limit"
)
return value
_GRAMMAR_PRIMITIVES = (
@@ -909,6 +936,7 @@ class UdfDefinition:
output_schema: Optional[pa.DataType | pa.Field | pa.Schema],
pip: tuple[str, ...],
env: Mapping[str, str],
secrets: tuple[str, ...],
python_version: Optional[str],
conda: tuple[str, ...] = (),
conda_channels: tuple[str, ...] = (),
@@ -935,6 +963,17 @@ class UdfDefinition:
for key, value in environment.items()
):
raise TypeError("Function env keys and values must be strings")
required_secrets = tuple(sorted(set(secrets)))
invalid_secrets = [
secret for secret in required_secrets if not _SECRET_NAME.fullmatch(secret)
]
if invalid_secrets:
raise ValueError(f"invalid Function secret names: {invalid_secrets!r}")
overlap = set(environment) & set(required_secrets)
if overlap:
raise ValueError(
f"Function env and secret names must be disjoint: {sorted(overlap)!r}"
)
signature = _infer_signature(function, input_schema, output_schema)
source = _package_source(function)
digest = f"sha256:{hashlib.sha256(source).hexdigest()}"
@@ -963,14 +1002,65 @@ class UdfDefinition:
),
signature=signature,
runtime=runtime,
required_secrets=required_secrets,
)
functools.update_wrapper(self, function)
@property
def registration_request(self) -> FunctionRegistrationRequest:
"""The immutable request sent by ``create_function_async``."""
"""The immutable, value-free client model for a Function submission."""
return self._request
def _submission_json(self, secrets: Optional[Mapping[str, str]]) -> str:
"""Build one registration submission without retaining values on self."""
if secrets is None:
secret_values: Mapping[str, str] = {}
elif not isinstance(secrets, Mapping):
raise TypeError("Function secrets must be a mapping of names to strings")
else:
secret_values = secrets
if any(not isinstance(name, str) for name in secret_values):
raise TypeError("Function secret names must be strings")
expected = set(self._request.required_secrets)
provided = set(secret_values)
if provided != expected:
missing = sorted(expected - provided)
unexpected = sorted(provided - expected)
details = []
if missing:
details.append(f"missing: {missing!r}")
if unexpected:
details.append(f"unexpected: {unexpected!r}")
raise ValueError(
"Function secret values must exactly match the declared secrets ("
+ "; ".join(details)
+ ")"
)
canonical_values = {}
total_bytes = 0
for name in sorted(secret_values):
value = _validate_secret_value(name, secret_values[name])
total_bytes += len(value.encode("utf-8"))
if total_bytes > _MAX_FUNCTION_SECRET_VALUES_BYTES:
raise ValueError(
"Function secret values exceed the "
f"{_MAX_FUNCTION_SECRET_VALUES_BYTES}-byte request limit"
)
canonical_values[name] = value
submission = self._request._known_dict()
if canonical_values:
submission["secret_values"] = canonical_values
return json.dumps(
submission,
ensure_ascii=False,
allow_nan=False,
sort_keys=True,
separators=(",", ":"),
)
def __call__(self, *args, **kwargs):
return self._function(*args, **kwargs)
@@ -988,6 +1078,7 @@ def udf(
output_schema: Optional[pa.DataType | pa.Field | pa.Schema] = None,
pip: tuple[str, ...] | list[str] = (),
env: Optional[Mapping[str, str]] = None,
secrets: tuple[str, ...] | list[str] = (),
python_version: Optional[str] = None,
conda: tuple[str, ...] | list[str] = (),
conda_channels: tuple[str, ...] | list[str] = (),
@@ -1002,6 +1093,7 @@ def udf(
output_schema: Optional[pa.DataType | pa.Field | pa.Schema] = None,
pip: tuple[str, ...] | list[str] = (),
env: Optional[Mapping[str, str]] = None,
secrets: tuple[str, ...] | list[str] = (),
python_version: Optional[str] = None,
conda: tuple[str, ...] | list[str] = (),
conda_channels: tuple[str, ...] | list[str] = (),
@@ -1032,7 +1124,10 @@ def udf(
conda_channels : sequence of str, optional
Conda channels in priority order; requires ``conda``.
env : mapping of str to str, optional
Environment variables included in the Function definition.
Non-secret environment variables. Use ``secrets`` for credentials.
secrets : sequence of str, optional
Names of secrets required by the callable. Supply their values separately
to ``create_function`` or ``create_function_async``.
python_version : str, optional
Remote Python major/minor version. Defaults to the client version.
@@ -1054,11 +1149,15 @@ def udf(
Examples
--------
>>> from lancedb import udf
>>> @udf(pip=["numpy==2.2.0"])
>>> @udf(pip=["numpy==2.2.0"], secrets=["MODEL_TOKEN"])
... def score(value: float) -> float:
... return value * 2
>>> score(1.5)
3.0
>>> db.create_function( # doctest: +SKIP
... score, secrets={"MODEL_TOKEN": "user-secret-value"}
... )
"""
def decorate(target: Callable[..., Any]) -> UdfDefinition:
@@ -1069,6 +1168,7 @@ def udf(
output_schema=output_schema,
pip=tuple(pip),
env={} if env is None else env,
secrets=tuple(secrets),
python_version=python_version,
conda=tuple(conda),
conda_channels=tuple(conda_channels),
+10 -3
View File
@@ -7,7 +7,7 @@ import json
import logging
from concurrent.futures import ThreadPoolExecutor
import sys
from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Union
from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Mapping, Optional, Union
from urllib.parse import urlparse
import warnings
@@ -742,8 +742,15 @@ class RemoteDBConnection(DBConnection):
return Job(self._conn.job(job_id))
@override
def create_function_async(self, definition: UdfDefinition) -> Job[FunctionVersion]:
return Job(LOOP.run(self._conn.create_function_async(definition)))
def create_function_async(
self,
definition: UdfDefinition,
*,
secrets: Optional[Mapping[str, str]] = None,
) -> Job[FunctionVersion]:
return Job(
LOOP.run(self._conn.create_function_async(definition, secrets=secrets))
)
@override
def get_function(self, name: str, *, version: str) -> FunctionVersion:
@@ -37,6 +37,21 @@ def job_result(name: str) -> dict:
return json.loads(fixture(name))["result"]
def assert_no_secret_values(value):
if isinstance(value, dict):
for key, child in value.items():
assert key not in {
"secret_value",
"secret_values",
"resolved_secret",
"resolved_secrets",
}
assert_no_secret_values(child)
elif isinstance(value, list):
for child in value:
assert_no_secret_values(child)
def test_public_function_values_are_in_api_reference():
docs = Path(__file__).parents[3] / "docs" / "src" / "python" / "python.md"
rendered = docs.read_text()
@@ -94,6 +109,7 @@ def test_function_version_identity_is_immutable_and_exact():
version = FunctionVersion.from_json(json.dumps(value))
assert version.name == "embed"
assert version.version == "fv_01K3EXACT"
assert version.required_secrets == ("HF_TOKEN",)
with pytest.raises((TypeError, ValueError)):
version.version = "fv_changed"
@@ -276,6 +292,15 @@ def test_refresh_result_rejects_non_u64_values(field):
RefreshColumnResult.from_json(json.dumps(value))
def test_canonical_client_values_contain_secret_names_only():
version = FunctionVersion.from_json(
json.dumps(job_result("remote_function_job.json"))
)
canonical = json.loads(version.to_canonical_json())
assert canonical["required_secrets"] == ["HF_TOKEN"]
assert_no_secret_values(canonical)
class _FunctionDeclarationInner:
def __init__(self):
self.calls = []
@@ -19,7 +19,13 @@ import pyarrow as pa
import pytest
import lancedb
from lancedb.functions import UdfDefinition, udf
from lancedb.functions import (
_MAX_FUNCTION_SECRET_VALUE_BYTES,
_MAX_FUNCTION_SECRET_VALUES_BYTES,
FunctionRegistrationRequest,
UdfDefinition,
udf,
)
THRESHOLD = 20
_CACHE = None
@@ -39,12 +45,28 @@ FIXTURES = (
@udf(
pip=["numpy>=2"],
env={"MODE": "test"},
secrets=["API_TOKEN"],
python_version="3.12",
)
def normalize_score(value: float) -> float:
return value / 100.0
def _assert_no_secret_values(value):
if isinstance(value, dict):
for key, child in value.items():
assert key not in {
"secret_value",
"secret_values",
"resolved_secret",
"resolved_secrets",
}
_assert_no_secret_values(child)
elif isinstance(value, list):
for child in value:
_assert_no_secret_values(child)
def test_scalar_udf_matches_shared_registration_golden_and_remains_callable():
assert isinstance(normalize_score, UdfDefinition)
assert normalize_score(25.0) == 0.25
@@ -59,6 +81,8 @@ def test_scalar_udf_matches_shared_registration_golden_and_remains_callable():
"kind": "scalar_to_arrow_batch",
"version": 1,
}
assert request["required_secrets"] == ["API_TOKEN"]
_assert_no_secret_values(request)
def _run_packaged(definition, *args):
@@ -372,6 +396,7 @@ def test_udf_recursion_versus_a_rebound_module_name(tmp_path):
output_schema=None,
pip=(),
env={},
secrets=(),
python_version=None,
)
with pytest.raises(ValueError, match="binds that name to another value"):
@@ -526,13 +551,54 @@ def test_annotation_and_explicit_schema_validation_fail_closed():
return value
def test_secret_names_are_canonical_and_disjoint_from_environment():
@udf(secrets=["Z_TOKEN", "A_TOKEN", "Z_TOKEN"])
def canonical_secrets(value: int) -> int:
return value
assert canonical_secrets.registration_request.required_secrets == (
"A_TOKEN",
"Z_TOKEN",
)
with pytest.raises(ValueError, match="must be disjoint"):
@udf(env={"TOKEN": "plaintext"}, secrets=["TOKEN"])
def overlapping(value: int) -> int:
return value
def test_declared_secret_api_still_requires_explicit_create_values():
@udf(secrets=["API_TOKEN"])
def declared_secret(value: int) -> int:
return value
with pytest.raises(ValueError, match="missing"):
declared_secret._submission_json(None)
submission = json.loads(
declared_secret._submission_json({"API_TOKEN": "explicit-secret"})
)
assert submission["required_secrets"] == ["API_TOKEN"]
assert submission["secret_values"] == {"API_TOKEN": "explicit-secret"}
def test_no_secrets_preserve_canonical_registration_shape():
@udf
def no_secrets(value: int) -> int:
return value
canonical = json.loads(no_secrets.registration_request.to_canonical_json())
assert "required_secrets" not in canonical
assert json.loads(no_secrets._submission_json(None)) == canonical
def test_local_function_catalog_operations_are_not_supported(tmp_path):
db = lancedb.connect(tmp_path)
message = "Function catalog operations are not supported by this database"
with pytest.raises(NotImplementedError, match=message):
db.create_function(normalize_score)
db.create_function(normalize_score, secrets={"API_TOKEN": "value"})
with pytest.raises(NotImplementedError, match=message):
db.create_function_async(normalize_score)
db.create_function_async(normalize_score, secrets={"API_TOKEN": "value"})
with pytest.raises(NotImplementedError, match=message):
db.get_function("normalize_score", version="fv_exact")
@@ -562,6 +628,7 @@ def _mock_remote_function_catalog():
"runtime": body["runtime"],
"runtime_digest": "sha256:runtime",
"environment_digest": "sha256:environment",
"required_secrets": body.get("required_secrets", []),
"created_at": "2026-08-21T00:00:00Z",
}
response = {"job_id": "job-register"}
@@ -608,7 +675,9 @@ def test_remote_registration_job_and_exact_version_reopen_round_trip():
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
registration = db.create_function_async(normalize_score)
registration = db.create_function_async(
normalize_score, secrets={"API_TOKEN": "secret-value"}
)
assert registration.id == "job-register"
created = registration.wait()
reopened = db.get_function("normalize_score", version=created.version)
@@ -617,9 +686,18 @@ def test_remote_registration_job_and_exact_version_reopen_round_trip():
assert reopened.name == "normalize_score"
assert reopened.version == "fv_exact"
create_request = state["requests"][0][1]
assert create_request == json.loads(
expected = json.loads(normalize_score.registration_request.to_canonical_json())
expected["secret_values"] = {"API_TOKEN": "secret-value"}
assert create_request == expected
durable_request = FunctionRegistrationRequest.from_json(json.dumps(create_request))
assert not hasattr(durable_request, "secret_values")
assert "secret_values" not in json.loads(durable_request.to_canonical_json())
assert "secret_values" not in json.loads(
normalize_score.registration_request.to_canonical_json()
)
assert "secret-value" not in repr(normalize_score)
assert "secret-value" not in repr(normalize_score.registration_request)
assert not hasattr(created, "secret_values")
def test_blocking_remote_registration_returns_function_version():
@@ -630,7 +708,9 @@ def test_blocking_remote_registration_returns_function_version():
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
created = db.create_function(normalize_score)
created = db.create_function(
normalize_score, secrets={"API_TOKEN": "blocking-secret"}
)
assert created.name == "normalize_score"
assert created.version == "fv_exact"
@@ -638,3 +718,121 @@ def test_blocking_remote_registration_returns_function_version():
"/v1/functions/create",
"/v1/jobs/describe",
]
assert state["requests"][0][1]["secret_values"] == {"API_TOKEN": "blocking-secret"}
@pytest.mark.parametrize(
("secret_values", "error_type", "message"),
[
(None, ValueError, "missing"),
({}, ValueError, "missing"),
({"OTHER": "value"}, ValueError, "missing.*unexpected"),
({"API_TOKEN": ""}, ValueError, "non-empty"),
({"API_TOKEN": "bad\0value"}, ValueError, "NUL"),
({"API_TOKEN": 123}, TypeError, "must be a string"),
([("API_TOKEN", "value")], TypeError, "must be a mapping"),
],
)
def test_secret_values_are_validated_before_remote_request(
secret_values, error_type, message
):
with _mock_remote_function_catalog() as (host, state):
db = lancedb.connect(
"db://dev",
api_key="fake",
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
with pytest.raises(error_type, match=message):
db.create_function_async(normalize_score, secrets=secret_values)
assert state["requests"] == []
@pytest.mark.parametrize(
"value",
[
"x" * _MAX_FUNCTION_SECRET_VALUE_BYTES,
"é" * (_MAX_FUNCTION_SECRET_VALUE_BYTES // len("é".encode("utf-8"))),
],
ids=["ascii", "multibyte"],
)
def test_secret_value_accepts_exact_utf8_byte_limit(value):
submission = json.loads(normalize_score._submission_json({"API_TOKEN": value}))
assert submission["secret_values"]["API_TOKEN"] == value
assert len(value.encode("utf-8")) == _MAX_FUNCTION_SECRET_VALUE_BYTES
@pytest.mark.parametrize(
"value",
[
"x" * (_MAX_FUNCTION_SECRET_VALUE_BYTES + 1),
"é" * (_MAX_FUNCTION_SECRET_VALUE_BYTES // len("é".encode("utf-8")) + 1),
],
ids=["ascii", "multibyte"],
)
def test_secret_value_rejects_over_utf8_byte_limit_before_json_construction(
monkeypatch, value
):
def fail_if_json_construction_starts(self):
pytest.fail("oversized secret reached JSON construction")
monkeypatch.setattr(
FunctionRegistrationRequest, "_known_dict", fail_if_json_construction_starts
)
with pytest.raises(ValueError, match=r"exceeds the 65536-byte limit"):
normalize_score._submission_json({"API_TOKEN": value})
def test_secret_values_accept_exact_aggregate_utf8_byte_limit(monkeypatch):
names = tuple(f"SECRET_{index}" for index in range(8))
value = "é" * (_MAX_FUNCTION_SECRET_VALUE_BYTES // len("é".encode("utf-8")))
values = {name: value for name in names}
monkeypatch.setattr(
normalize_score,
"_request",
normalize_score._request._copy(update={"required_secrets": names}),
)
submission = json.loads(normalize_score._submission_json(values))
assert submission["secret_values"] == values
assert sum(len(item.encode("utf-8")) for item in values.values()) == (
_MAX_FUNCTION_SECRET_VALUES_BYTES
)
def test_secret_values_reject_aggregate_over_limit_before_construction(monkeypatch):
names = tuple(f"SECRET_{index}" for index in range(9))
values = {name: "x" * _MAX_FUNCTION_SECRET_VALUE_BYTES for name in names}
monkeypatch.setattr(
normalize_score,
"_request",
normalize_score._request._copy(update={"required_secrets": names}),
)
def fail_if_json_construction_starts(self):
pytest.fail("oversized aggregate reached JSON construction")
monkeypatch.setattr(
FunctionRegistrationRequest, "_known_dict", fail_if_json_construction_starts
)
with pytest.raises(ValueError, match=r"exceed.*524288-byte request limit"):
normalize_score._submission_json(values)
@pytest.mark.asyncio
async def test_async_remote_registration_submits_secret_values_only_once():
with _mock_remote_function_catalog() as (host, state):
db = await lancedb.connect_async(
"db://dev",
api_key="fake",
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
registration = await db.create_function_async(
normalize_score, secrets={"API_TOKEN": "async-secret"}
)
created = await registration.wait()
assert state["requests"][0][1]["secret_values"] == {"API_TOKEN": "async-secret"}
assert not hasattr(created, "secret_values")
+309 -4
View File
@@ -5,9 +5,9 @@
//! backend-neutral terminal result of a computed-column refresh.
//!
//! This module contains client/wire values only. Catalog persistence,
//! environment bake, and execution are owned by Sophon.
//! environment bake, secret resolution, and execution are owned by Sophon.
use std::collections::BTreeMap;
use std::collections::{BTreeMap, BTreeSet};
use serde::de::{self, DeserializeOwned};
use serde::{Deserialize, Deserializer, Serialize, Serializer};
@@ -15,6 +15,16 @@ use serde_json::Value;
use crate::{Error, Result};
// Keep these byte limits aligned with Sophon's Function submission validation.
pub(crate) const MAX_FUNCTION_SECRET_VALUE_BYTES: usize = 64 * 1024;
const MAX_FUNCTION_SECRET_VALUES_BYTES: usize = 512 * 1024;
fn is_portable_environment_name(name: &str) -> bool {
let mut bytes = name.bytes();
matches!(bytes.next(), Some(b'A'..=b'Z' | b'a'..=b'z' | b'_'))
&& bytes.all(|byte| matches!(byte, b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'_'))
}
fn invalid_json(error: impl std::fmt::Display) -> Error {
Error::InvalidInput {
message: format!("invalid remote Function JSON: {error}"),
@@ -198,6 +208,11 @@ pub struct PythonEnvironmentSpec {
}
/// Reproducible Python runtime definition understood by Sophon.
///
/// `env` contains non-secret values. Secret values are submission-only in the
/// client model and do not become part of this public runtime identity;
/// [`FunctionVersion::required_secrets`] contains names only. Sophon persists
/// submitted values separately in the private execution artifact.
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum PythonRuntimeSpec {
@@ -239,7 +254,7 @@ impl PythonRuntimeSpec {
}
}
/// Environment variables, or `None` for an unknown kind.
/// Non-secret environment variables, or `None` for an unknown kind.
pub fn env(&self) -> Option<&BTreeMap<String, String>> {
match self {
Self::Python { env, .. } => Some(env),
@@ -324,6 +339,8 @@ pub struct FunctionVersion {
runtime: PythonRuntimeSpec,
runtime_digest: String,
environment_digest: String,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
required_secrets: Vec<String>,
created_at: String,
}
@@ -356,6 +373,12 @@ impl FunctionVersion {
&self.environment_digest
}
/// Required secret names. Resolved values exist only in Sophon's private
/// execution artifact and worker launch path.
pub fn required_secrets(&self) -> &[String] {
&self.required_secrets
}
pub fn created_at(&self) -> &str {
&self.created_at
}
@@ -397,12 +420,115 @@ pub struct FunctionArtifactRequest {
}
/// Stable request envelope for remote immutable Function registration.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
///
/// Secret values are submission-only in the client model. Sophon persists them
/// in the database-scoped private execution artifact; returned
/// [`FunctionVersion`] and Job metadata contain only
/// [`Self::required_secrets`] names. Debug formatting always redacts values.
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct FunctionRegistrationRequest {
pub name: String,
pub artifact: FunctionArtifactRequest,
pub signature: FunctionSignature,
pub runtime: PythonRuntimeSpec,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub required_secrets: Vec<String>,
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
pub secret_values: BTreeMap<String, String>,
}
impl FunctionRegistrationRequest {
pub(crate) fn validate_secret_values(&self) -> Result<()> {
let mut required = BTreeSet::new();
for name in &self.required_secrets {
if !is_portable_environment_name(name) {
return Err(Error::InvalidInput {
message: format!(
"Function secret name {name:?} must be a portable environment variable name"
),
});
}
if !required.insert(name) {
return Err(Error::InvalidInput {
message: format!("Function required_secrets contains duplicate name {name:?}"),
});
}
}
if let PythonRuntimeSpec::Python { env, .. } = &self.runtime
&& let Some(name) = required.iter().find(|name| env.contains_key(**name))
{
return Err(Error::InvalidInput {
message: format!(
"Function runtime env and secret names must be disjoint: {name:?}"
),
});
}
let provided = self.secret_values.keys().collect::<BTreeSet<_>>();
if required != provided {
return Err(Error::InvalidInput {
message: "Function secret_values keys must exactly match required_secrets"
.to_string(),
});
}
let mut total_bytes = 0usize;
for (name, value) in &self.secret_values {
if value.is_empty() {
return Err(Error::InvalidInput {
message: format!("Function secret {name:?} value must be non-empty"),
});
}
if value.contains('\0') {
return Err(Error::InvalidInput {
message: format!("Function secret {name:?} value must not contain NUL"),
});
}
if value.len() > MAX_FUNCTION_SECRET_VALUE_BYTES {
return Err(Error::InvalidInput {
message: format!(
"Function secret {name:?} value exceeds the \
{MAX_FUNCTION_SECRET_VALUE_BYTES}-byte limit"
),
});
}
total_bytes =
total_bytes
.checked_add(value.len())
.ok_or_else(|| Error::InvalidInput {
message: "Function secret values exceed the request byte limit".to_string(),
})?;
}
if total_bytes > MAX_FUNCTION_SECRET_VALUES_BYTES {
return Err(Error::InvalidInput {
message: format!(
"Function secret values exceed the \
{MAX_FUNCTION_SECRET_VALUES_BYTES}-byte request limit"
),
});
}
Ok(())
}
}
impl std::fmt::Debug for FunctionRegistrationRequest {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let secret_values = self
.secret_values
.keys()
.map(|name| (name, "[REDACTED]"))
.collect::<BTreeMap<_, _>>();
formatter
.debug_struct("FunctionRegistrationRequest")
.field("name", &self.name)
.field("artifact", &self.artifact)
.field("signature", &self.signature)
.field("runtime", &self.runtime)
.field("required_secrets", &self.required_secrets)
.field("secret_values", &secret_values)
.finish()
}
}
impl_json!(FunctionRegistrationRequest);
@@ -587,6 +713,185 @@ impl RefreshColumnResult {
impl_json!(RefreshColumnResult);
#[cfg(test)]
mod secret_value_tests {
use super::{
FunctionRegistrationRequest, MAX_FUNCTION_SECRET_VALUE_BYTES,
MAX_FUNCTION_SECRET_VALUES_BYTES, PythonRuntimeSpec,
};
use crate::Error;
fn request() -> FunctionRegistrationRequest {
FunctionRegistrationRequest::from_json(include_str!(
"../tests/fixtures/first_class_functions/v1/remote_function_registration_request.json"
))
.unwrap()
}
#[test]
fn validates_secret_name_and_value_invariants() {
let missing = request();
assert!(matches!(
missing.validate_secret_values(),
Err(Error::InvalidInput { message }) if message.contains("exactly match")
));
let mut empty = request();
empty
.secret_values
.insert("API_TOKEN".to_string(), String::new());
assert!(matches!(
empty.validate_secret_values(),
Err(Error::InvalidInput { message }) if message.contains("non-empty")
));
let mut nul = request();
nul.secret_values
.insert("API_TOKEN".to_string(), "before\0after".to_string());
assert!(matches!(
nul.validate_secret_values(),
Err(Error::InvalidInput { message }) if message.contains("NUL")
));
let mut unexpected = request();
unexpected
.secret_values
.insert("OTHER".to_string(), "value".to_string());
assert!(matches!(
unexpected.validate_secret_values(),
Err(Error::InvalidInput { message }) if message.contains("exactly match")
));
}
#[test]
fn rejects_invalid_duplicate_and_overlapping_secret_declarations() {
let mut invalid_name = request();
invalid_name.required_secrets = vec!["BAD=NAME".to_string()];
invalid_name
.secret_values
.insert("BAD=NAME".to_string(), "secret".to_string());
let mut duplicate = request();
duplicate.required_secrets = vec!["API_TOKEN".to_string(), "API_TOKEN".to_string()];
duplicate
.secret_values
.insert("API_TOKEN".to_string(), "secret".to_string());
let mut overlap = request();
overlap
.secret_values
.insert("API_TOKEN".to_string(), "secret".to_string());
if let PythonRuntimeSpec::Python { env, .. } = &mut overlap.runtime {
env.insert("API_TOKEN".to_string(), "public".to_string());
}
assert!(matches!(
invalid_name.validate_secret_values(),
Err(Error::InvalidInput { message }) if message.contains("portable environment variable")
));
assert!(matches!(
duplicate.validate_secret_values(),
Err(Error::InvalidInput { message }) if message.contains("duplicate")
));
assert!(matches!(
overlap.validate_secret_values(),
Err(Error::InvalidInput { message }) if message.contains("must be disjoint")
));
}
#[test]
fn enforces_portable_secret_name_boundaries() {
for name in ["A", "_", "A0_"] {
let mut request = request();
request.required_secrets = vec![name.to_string()];
request
.secret_values
.insert(name.to_string(), "secret".to_string());
request.validate_secret_values().unwrap();
}
for name in ["", "0TOKEN", "BAD-NAME", "TÖKEN"] {
let mut request = request();
request.required_secrets = vec![name.to_string()];
request
.secret_values
.insert(name.to_string(), "secret".to_string());
assert!(matches!(
request.validate_secret_values(),
Err(Error::InvalidInput { message })
if message.contains("portable environment variable")
));
}
}
#[test]
fn accepts_exact_secret_value_utf8_byte_limit() {
for value in [
"x".repeat(MAX_FUNCTION_SECRET_VALUE_BYTES),
"é".repeat(MAX_FUNCTION_SECRET_VALUE_BYTES / "é".len()),
] {
assert_eq!(value.len(), MAX_FUNCTION_SECRET_VALUE_BYTES);
let mut request = request();
request.secret_values.insert("API_TOKEN".to_string(), value);
request.validate_secret_values().unwrap();
}
}
#[test]
fn rejects_secret_value_over_utf8_byte_limit() {
for value in [
"x".repeat(MAX_FUNCTION_SECRET_VALUE_BYTES + 1),
"é".repeat(MAX_FUNCTION_SECRET_VALUE_BYTES / "é".len() + 1),
] {
assert!(value.len() > MAX_FUNCTION_SECRET_VALUE_BYTES);
let mut request = request();
request.secret_values.insert("API_TOKEN".to_string(), value);
assert!(matches!(
request.validate_secret_values(),
Err(Error::InvalidInput { message }) if message.contains("65536-byte limit")
));
}
}
#[test]
fn rejects_aggregate_secret_value_bytes_over_server_limit() {
let mut request = request();
request.required_secrets = (0..9).map(|index| format!("SECRET_{index}")).collect();
request.secret_values = request
.required_secrets
.iter()
.map(|name| (name.clone(), "x".repeat(MAX_FUNCTION_SECRET_VALUE_BYTES)))
.collect();
assert!(matches!(
request.validate_secret_values(),
Err(Error::InvalidInput { message })
if message.contains(&format!("{MAX_FUNCTION_SECRET_VALUES_BYTES}-byte request limit"))
));
}
#[test]
fn accepts_exact_aggregate_secret_value_byte_limit() {
let mut request = request();
request.required_secrets = (0..8).map(|index| format!("SECRET_{index}")).collect();
request.secret_values = request
.required_secrets
.iter()
.map(|name| (name.clone(), "x".repeat(MAX_FUNCTION_SECRET_VALUE_BYTES)))
.collect();
assert_eq!(
request
.secret_values
.values()
.map(String::len)
.sum::<usize>(),
MAX_FUNCTION_SECRET_VALUES_BYTES
);
request.validate_secret_values().unwrap();
}
}
#[cfg(test)]
mod conda_environment_tests {
use super::PythonEnvironmentSpec;
+100 -15
View File
@@ -7,6 +7,7 @@ use reqwest::{
Body, Request, RequestBuilder, Response,
header::{HeaderMap, HeaderValue},
};
use serde_json::Value;
use std::{collections::HashMap, future::Future, str::FromStr, sync::Arc, time::Duration};
use crate::error::{Error, Result};
@@ -14,6 +15,60 @@ use crate::remote::db::RemoteOptions;
use crate::remote::retry::{ResolvedRetryConfig, RetryCounter};
const REQUEST_ID_HEADER: HeaderName = HeaderName::from_static("x-request-id");
const REDACTED_JSON_VALUE: &str = "[REDACTED]";
const SUPPRESSED_JSON_BODY: &str = "[JSON BODY SUPPRESSED]";
fn is_sensitive_json_field(name: &str) -> bool {
name.to_ascii_lowercase().contains("secret")
}
fn redact_sensitive_json_fields(value: &mut Value) {
match value {
Value::Object(fields) => {
for (name, child) in fields {
if is_sensitive_json_field(name) {
*child = Value::String(REDACTED_JSON_VALUE.to_string());
} else {
redact_sensitive_json_fields(child);
}
}
}
Value::Array(values) => values.iter_mut().for_each(redact_sensitive_json_fields),
_ => {}
}
}
fn redacted_json_body(request: &Request) -> Option<String> {
let body = request.body()?.as_bytes()?;
let mut value = serde_json::from_slice(body).ok()?;
redact_sensitive_json_fields(&mut value);
serde_json::to_string(&value).ok()
}
fn request_log_message(request: &Request, request_id: &str) -> String {
let prefix = format!(
"Sending request_id={}: {} {}",
request_id,
request.method(),
request.url()
);
let content_type = request
.headers()
.get("content-type")
.and_then(|value| value.to_str().ok())
.and_then(|value| value.split(';').next());
if content_type.is_some_and(|value| value.eq_ignore_ascii_case("application/json")) {
// Never format the raw Request here: its Debug representation is not a
// redaction boundary and may include the original body. If the JSON body
// cannot be structurally parsed, suppress it instead of logging raw bytes.
let body = redacted_json_body(request).unwrap_or_else(|| SUPPRESSED_JSON_BODY.to_string());
format!("{prefix} with body {body}")
} else {
// Method and URL are sufficient request context. Raw Request formatting
// may expose headers or a non-JSON body, so it is never a logging fallback.
prefix
}
}
/// Configuration for TLS/mTLS settings.
#[derive(Clone, Debug)]
@@ -839,22 +894,9 @@ impl<S: HttpSend> RestfulLanceDbClient<S> {
}
}
pub(crate) fn log_request(&self, request: &Request, request_id: &String) {
pub(crate) fn log_request(&self, request: &Request, request_id: &str) {
if log::log_enabled!(log::Level::Debug) {
let content_type = request
.headers()
.get("content-type")
.map(|v| v.to_str().unwrap());
if content_type == Some("application/json") {
let body = request.body().as_ref().unwrap().as_bytes().unwrap();
let body = String::from_utf8_lossy(body);
debug!(
"Sending request_id={}: {:?} with body {}",
request_id, request, body
);
} else {
debug!("Sending request_id={}: {:?}", request_id, request);
}
debug!("{}", request_log_message(request, request_id));
}
}
@@ -1077,6 +1119,49 @@ mod tests {
ENV_MUTEX.lock().unwrap_or_else(|e| e.into_inner())
}
#[test]
fn test_request_log_message_redacts_secrets_and_never_formats_raw_requests() {
const SECRET_SENTINEL: &str = "udf-secret-log-sentinel-7e4e";
const MALFORMED_SENTINEL: &str = "malformed-secret-log-sentinel-b652";
const NON_JSON_SENTINEL: &str = "non-json-secret-log-sentinel-7fd1";
let request = reqwest::Client::new()
.post("https://example.com/v1/functions/create")
.json(&serde_json::json!({
"name": "uses_secret",
"nested": {
"secret_values": {"OPENAI_API_KEY": SECRET_SENTINEL},
"safe": "visible-value"
}
}))
.build()
.unwrap();
let log_message = request_log_message(&request, "valid-json");
let malformed_request = reqwest::Client::new()
.post("https://example.com/v1/functions/create")
.header("content-type", "application/json; charset=utf-8")
.body(format!(r#"{{"secret_values":"{MALFORMED_SENTINEL}""#))
.build()
.unwrap();
let malformed_log_message = request_log_message(&malformed_request, "malformed-json");
let non_json_request = reqwest::Client::new()
.post("https://example.com/v1/functions/create")
.header("content-type", "text/plain")
.body(NON_JSON_SENTINEL)
.build()
.unwrap();
let non_json_log_message = request_log_message(&non_json_request, "non-json");
assert!(log_message.contains("visible-value"));
assert!(log_message.contains(REDACTED_JSON_VALUE));
assert!(!log_message.contains(SECRET_SENTINEL));
assert!(malformed_log_message.contains(SUPPRESSED_JSON_BODY));
assert!(!malformed_log_message.contains(MALFORMED_SENTINEL));
assert!(!non_json_log_message.contains(NON_JSON_SENTINEL));
}
#[test]
fn test_timeout_config_default() {
let config = TimeoutConfig::default();
+33 -2
View File
@@ -554,6 +554,7 @@ impl<S: HttpSend> Database for RemoteDatabase<S> {
&self,
request: FunctionRegistrationRequest,
) -> Result<Job<FunctionVersion>> {
request.validate_secret_values()?;
let req = self.client.post("/v1/functions/create").json(&request);
let (request_id, response) = self.client.send(req).await?;
let response = self.client.check_response(&request_id, response).await?;
@@ -2642,7 +2643,8 @@ mod tests {
);
const FUNCTION_JOB: &str =
include_str!("../../tests/fixtures/first_class_functions/v1/remote_function_job.json");
let expected: serde_json::Value = serde_json::from_str(REQUEST).unwrap();
let mut expected: serde_json::Value = serde_json::from_str(REQUEST).unwrap();
expected["secret_values"] = serde_json::json!({"API_TOKEN": "secret-value"});
let conn = Connection::new_with_handler(move |request| match request.url().path() {
"/v1/functions/create" => {
assert_eq!(request.method(), &reqwest::Method::POST);
@@ -2660,7 +2662,10 @@ mod tests {
.unwrap(),
path => panic!("unexpected path: {path}"),
});
let request = crate::function::FunctionRegistrationRequest::from_json(REQUEST).unwrap();
let mut request = crate::function::FunctionRegistrationRequest::from_json(REQUEST).unwrap();
request
.secret_values
.insert("API_TOKEN".to_string(), "secret-value".to_string());
let job = conn.create_function_async(request).await.unwrap();
assert_eq!(job.id(), Some("job-function-1"));
let version = job.wait().await.unwrap();
@@ -2668,6 +2673,32 @@ mod tests {
assert_eq!(version.version(), "fv_01K3EXACT");
}
#[tokio::test]
async fn test_create_function_async_validates_secrets_before_serialization_and_send() {
const REQUEST: &str = include_str!(
"../../tests/fixtures/first_class_functions/v1/remote_function_registration_request.json"
);
let sends = Arc::new(AtomicUsize::new(0));
let sends_ref = sends.clone();
let conn = Connection::new_with_handler(move |_| {
sends_ref.fetch_add(1, Ordering::SeqCst);
http::Response::builder().status(500).body("").unwrap()
});
let mut request = crate::function::FunctionRegistrationRequest::from_json(REQUEST).unwrap();
request.secret_values.insert(
"API_TOKEN".to_string(),
"x".repeat(crate::function::MAX_FUNCTION_SECRET_VALUE_BYTES + 1),
);
let error = conn.create_function_async(request).await.unwrap_err();
assert!(matches!(
error,
Error::InvalidInput { message } if message.contains("65536-byte limit")
));
assert_eq!(sends.load(Ordering::SeqCst), 0);
}
#[tokio::test]
async fn test_get_function_requires_and_sends_exact_version() {
const VERSION: &str = include_str!(
+2 -3
View File
@@ -5734,9 +5734,8 @@ mod tests {
))
.execute()
.await;
let err = match result {
Ok(_) => panic!("legacy remote query unexpectedly succeeded"),
Err(err) => err,
let Err(err) = result else {
panic!("legacy remote query unexpectedly succeeded")
};
assert!(
+97 -8
View File
@@ -22,7 +22,7 @@
use std::collections::{BTreeSet, HashMap};
use std::sync::Arc;
use arrow_schema::{DataType, Field as ArrowField, Schema as ArrowSchema, SchemaRef};
use arrow_schema::{DataType, Field as ArrowField, Fields, Schema as ArrowSchema, SchemaRef};
use datafusion_common::tree_node::TreeNode;
use datafusion_physical_plan::PhysicalExpr;
use lance::dataset::NewColumnTransform;
@@ -1273,6 +1273,11 @@ pub(crate) fn bind(schema: SchemaRef, column: &str, expression: &str) -> Result<
/// refresh time: that the expression parses, that every column it reads
/// exists, and that the target name is free. A declaration that survives this
/// is one a refresh can always act on.
///
/// Each accepted column joins the schema the next one resolves against, so a
/// batch may declare `a` and then `b = a + 1` in one commit. Refresh fills a
/// column's computed inputs before the column, so the order of refresh calls
/// does not matter.
pub(crate) fn plan(schema: SchemaRef, columns: &[(String, String)]) -> Result<Vec<ArrowField>> {
if columns.is_empty() {
return Err(Error::InvalidInput {
@@ -1280,11 +1285,11 @@ pub(crate) fn plan(schema: SchemaRef, columns: &[(String, String)]) -> Result<Ve
});
}
let mut schema = schema;
let mut fields = Vec::with_capacity(columns.len());
let mut declared: Vec<&str> = Vec::with_capacity(columns.len());
for (name, expression) in columns {
if schema.field_with_name(name).is_ok() || declared.contains(&name.as_str()) {
if schema.field_with_name(name).is_ok() {
return Err(Error::ColumnAlreadyExists { name: name.clone() });
}
@@ -1292,16 +1297,50 @@ pub(crate) fn plan(schema: SchemaRef, columns: &[(String, String)]) -> Result<Ve
// Declared columns start entirely null, so nullability is a property
// of the declaration rather than of what the expression yields.
fields.push(
ArrowField::new(name, bound.data_type, true)
.with_metadata(computed_column_metadata(expression, &bound.inputs)),
);
declared.push(name);
let field = ArrowField::new(name, bound.data_type, true)
.with_metadata(computed_column_metadata(expression, &bound.inputs));
schema = Arc::new(ArrowSchema::new_with_metadata(
schema
.fields()
.iter()
.cloned()
.chain(std::iter::once(Arc::new(field.clone())))
.collect::<Fields>(),
schema.metadata().clone(),
));
fields.push(field);
}
Ok(fields)
}
/// Run the schema-level checks of
/// [`AddColumnsBuilder::computed`](super::AddColumnsBuilder::computed) against
/// `schema` without committing: the Function-binding guard and the planning of
/// every declaration. For callers that stage declarations behind other work
/// and need those rejections before any of it lands.
///
/// Only the schema is consulted. Declaring also refuses a table with an LSM
/// write spec or retained SSTables; that is table state, checked at commit.
///
/// ```
/// # use std::sync::Arc;
/// # use arrow_schema::{DataType, Field, Schema};
/// use lancedb::table::computed_columns::validate_declarations;
///
/// let schema = Arc::new(Schema::new(vec![Field::new("x", DataType::Int32, false)]));
/// let declarations = vec![
/// ("a".to_string(), "x + 1".to_string()),
/// ("b".to_string(), "a * 2".to_string()),
/// ];
/// assert!(validate_declarations(schema.clone(), &declarations).is_ok());
/// assert!(validate_declarations(schema, &[("c".into(), "random()".into())]).is_err());
/// ```
pub fn validate_declarations(schema: SchemaRef, columns: &[(String, String)]) -> Result<()> {
ensure_no_function_bindings_for_mutation(schema.as_ref(), "schema evolution")?;
plan(schema, columns).map(drop)
}
/// Build the transform that declares `columns` against `schema`.
///
/// An all-null column is how a binding with no values yet is carried into a
@@ -1340,6 +1379,22 @@ pub(super) async fn add_foreign_kind(table: &crate::Table, name: &str, kind: &st
#[cfg(test)]
mod tests {
/// The gate's reproducer: the validator applies the same schema-level
/// guard declaring does, so a staging caller is refused before it commits
/// anything else.
#[test]
fn test_validate_declarations_matches_schema_admission_barriers() {
let schema = Arc::new(ArrowSchema::new_with_metadata(
vec![ArrowField::new("x", DataType::Int32, true)],
HashMap::from([(
FUNCTION_BINDINGS_META_KEY.to_string(),
"not valid binding metadata".to_string(),
)]),
));
let declarations = vec![("a".to_string(), "x + 1".to_string())];
assert!(super::validate_declarations(schema, &declarations).is_err());
}
#[test]
fn output_arrow_type_grammar_matches_the_shared_golden() {
let golden: serde_json::Value = serde_json::from_str(include_str!(
@@ -1582,6 +1637,40 @@ mod tests {
assert!(declared(&table).await.is_empty());
}
/// A batch may build on itself: one commit, and the later entry's inputs
/// name the earlier one.
#[tokio::test]
async fn test_a_declaration_may_read_one_declared_before_it() {
let table = table_with_ints("chain").await;
let before = table.version().await.unwrap();
add_computed(
&table,
&[("a".into(), "x + 1".into()), ("b".into(), "a * 2".into())],
)
.await
.unwrap();
assert_eq!(table.version().await.unwrap(), before + 1);
let declared = declared(&table).await;
assert_eq!(declared[1].name, "b");
assert_eq!(declared[1].inputs, vec!["a".to_string()]);
// Order is the dependency order; reading ahead is still unknown.
let err = add_computed(
&table,
&[("c".into(), "d + 1".into()), ("d".into(), "x + 1".into())],
)
.await
.unwrap_err();
assert!(matches!(err, Error::InvalidExpression { column, .. } if column == "c"));
assert!(
validate_declarations(
table.schema().await.unwrap(),
&[("e".into(), "random()".into())]
)
.is_err()
);
}
/// A column added by an ordinary transform is materialized, not bound, so
/// it carries no declaration to report.
#[tokio::test]
+363 -24
View File
@@ -7,6 +7,16 @@
//! therefore idempotent and does not observe input mutation -- once a row is
//! filled, changing what the expression reads leaves the stored result alone.
//!
//! A column's computed inputs are filled first -- the dependency graph is
//! walked once, each reachable column filled once in dependency order, each
//! fill its own commit. Every fill in the pass, the requested column's
//! included, covers only the fragments of the snapshot the pass started
//! from: a commit may rebase over a concurrent append, and the fragment that
//! admits carries placeholder nulls no earlier fill covered, so it waits for
//! a later refresh rather than being read as values. Two concurrent fills of
//! one input collide on its field in lance's conflict check, so a dependent
//! fill can only commit over inputs that were durable when it read them.
//!
//! Two passes per fragment. The first scans only the unfilled live rows and
//! evaluates the expression over them, which yields the exact fill count and
//! decides whether the fragment is staged at all -- a fragment where nothing
@@ -19,6 +29,7 @@
//! inputs masked to null first, so a poison value in a row nobody is filling
//! cannot fail the refresh.
use std::collections::HashSet;
use std::sync::Arc;
use arrow_array::{ArrayRef, BooleanArray, RecordBatch, RecordBatchOptions};
@@ -41,7 +52,8 @@ use crate::{Error, Result};
/// The result of refreshing a computed column.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Default)]
pub struct RefreshColumnResult {
/// Rows that had a value computed.
/// Rows that had a value computed, in the requested column only; inputs
/// filled on its behalf are not counted.
#[serde(default)]
pub rows_filled: u64,
/// The commit version associated with the operation.
@@ -51,7 +63,79 @@ pub struct RefreshColumnResult {
struct RefreshExecution {
result: RefreshColumnResult,
/// The snapshot the requested column was evaluated against, after its
/// inputs were filled.
source_version: u64,
/// The last version any fill in the pass committed, inputs included.
published_version: Option<u64>,
/// Unfilled live rows of the requested column in fragments the pass did
/// not cover, counted on the snapshot the result reports -- the published
/// one, or the source when nothing was published.
rows_deferred: u64,
}
/// One column's fill against one snapshot.
struct Fill {
rows_filled: u64,
/// The dataset the commit produced, if anything was filled.
committed: Option<Arc<Dataset>>,
}
/// Test-only one-shot pauses before a named column's fill:
/// `(column, reached, resume)`, one per column, consumed when hit.
#[cfg(test)]
type Pause = (String, Arc<tokio::sync::Notify>, Arc<tokio::sync::Notify>);
#[cfg(test)]
static PAUSES_BEFORE_FILL: std::sync::Mutex<Vec<Pause>> = std::sync::Mutex::new(Vec::new());
/// The SQL-computed columns `column` reads, transitively, each once, in an
/// order that fills every column after the columns it reads.
///
/// Declarations are acyclic by construction: a column can only read what
/// existed when it was declared.
fn dependency_order(schema: &ArrowSchema, column: &str) -> Result<Vec<String>> {
fn visit(
schema: &ArrowSchema,
column: &str,
visited: &mut HashSet<String>,
order: &mut Vec<String>,
) -> Result<()> {
let Some(declaration) = schema
.field_with_name(column)
.ok()
.and_then(computed_column_from_field)
else {
return Ok(());
};
for input in &declaration.inputs {
let input = super::computed_columns::root(input);
if visited.contains(input) {
continue;
}
let Some(input_declaration) = schema
.field_with_name(input)
.ok()
.and_then(computed_column_from_field)
else {
continue;
};
if !matches!(input_declaration.kind, ComputedColumnKind::Sql { .. }) {
return Err(Error::NotSupported {
message: format!(
"computed column '{column}' reads '{input}', which this refresh \
cannot fill first; refresh '{input}' before '{column}'"
),
});
}
visit(schema, input, visited, order)?;
visited.insert(input.to_string());
order.push(input.to_string());
}
Ok(())
}
let mut order = Vec::new();
visit(schema, column, &mut HashSet::new(), &mut order)?;
Ok(order)
}
/// Internal implementation of the refresh logic.
@@ -70,9 +154,96 @@ async fn execute_refresh_column_with_source(
) -> Result<RefreshExecution> {
table.dataset.ensure_mutable()?;
ensure_no_lsm_write_spec(table).await?;
let dataset = table.dataset.get().await?;
let mut dataset = table.dataset.get().await?;
declared_expression(&dataset, column)?;
let expression = declared_expression(&dataset, column)?;
// The pass covers exactly these fragments. A commit below may rebase over
// a concurrent append, and the fragment that admits was scanned by no
// earlier fill, so it is excluded from every later one.
let eligible: HashSet<u64> = dataset
.get_fragments()
.iter()
.map(|f| f.id() as u64)
.collect();
let mut published_version = None;
for input in dependency_order(&ArrowSchema::from(dataset.schema()), column)? {
pause_before_fill(&input).await;
if let Some(committed) = fill_column(table, &dataset, &input, &eligible)
.await?
.committed
{
published_version = Some(committed.version().version);
dataset = committed;
}
}
pause_before_fill(column).await;
let source_version = dataset.version().version;
let fill = fill_column(table, &dataset, column, &eligible).await?;
if let Some(committed) = &fill.committed {
published_version = Some(committed.version().version);
}
// Counted after the commit: it may have rebased over an append, and the
// fragment that admits is deferred but part of the published version.
let terminal = fill.committed.as_ref().unwrap_or(&dataset);
let rows_deferred = count_deferred(terminal, column, &eligible).await?;
Ok(RefreshExecution {
result: RefreshColumnResult {
rows_filled: fill.rows_filled,
version: published_version.unwrap_or(source_version),
},
source_version,
published_version,
rows_deferred,
})
}
/// Unfilled live rows of `column` in `dataset`'s fragments outside `eligible`.
async fn count_deferred(dataset: &Dataset, column: &str, eligible: &HashSet<u64>) -> Result<u64> {
let deferred: Vec<_> = dataset
.get_fragments()
.into_iter()
.filter(|fragment| !eligible.contains(&(fragment.id() as u64)))
.map(|fragment| fragment.metadata().clone())
.collect();
if deferred.is_empty() {
return Ok(0);
}
let mut scanner = dataset.scan();
scanner
.with_fragments(deferred)
.filter(&format!("{} IS NULL", quote_identifier(column)))?;
Ok(scanner.count_rows().await?)
}
#[cfg(test)]
async fn pause_before_fill(column: &str) {
let pause = {
let mut pauses = PAUSES_BEFORE_FILL.lock().unwrap();
pauses
.iter()
.position(|(paused, _, _)| paused == column)
.map(|index| pauses.remove(index))
};
if let Some((_, reached, resume)) = pause {
reached.notify_one();
resume.notified().await;
}
}
#[cfg(not(test))]
async fn pause_before_fill(_column: &str) {}
/// Fill `column`'s unfilled live rows in the `eligible` fragments as they
/// stand in `dataset`, committing against that snapshot.
async fn fill_column(
table: &NativeTable,
dataset: &Arc<Dataset>,
column: &str,
eligible: &HashSet<u64>,
) -> Result<Fill> {
let expression = declared_expression(dataset, column)?;
let schema = Arc::new(ArrowSchema::from(dataset.schema()));
let bound = Arc::new(super::computed_columns::bind(schema, column, &expression)?);
let field = dataset
@@ -91,23 +262,22 @@ async fn execute_refresh_column_with_source(
let mut rows_filled = 0u64;
let mut replacements = Vec::new();
for fragment in dataset.get_fragments() {
let gained = count_fragment_gains(&dataset, &fragment, &bound, column).await?;
if !eligible.contains(&(fragment.id() as u64)) {
continue;
}
let gained = count_fragment_gains(dataset, &fragment, &bound, column).await?;
if gained == 0 {
continue;
}
rows_filled += gained;
let values = fill_stream(&dataset, &fragment, bound.clone(), column).await?;
let values = fill_stream(dataset, &fragment, bound.clone(), column).await?;
replacements.push(fragment.write_columns(values, &column_schema).await?);
}
if replacements.is_empty() {
let source_version = dataset.version().version;
return Ok(RefreshExecution {
result: RefreshColumnResult {
rows_filled: 0,
version: source_version,
},
source_version,
return Ok(Fill {
rows_filled: 0,
committed: None,
});
}
@@ -125,15 +295,10 @@ async fn execute_refresh_column_with_source(
false,
)
.await?;
let version = new_dataset.version().version;
table.dataset.update(new_dataset);
Ok(RefreshExecution {
result: RefreshColumnResult {
rows_filled,
version,
},
source_version: read_version,
table.dataset.update(new_dataset.clone());
Ok(Fill {
rows_filled,
committed: Some(Arc::new(new_dataset)),
})
}
@@ -158,10 +323,9 @@ pub(crate) async fn execute_refresh_column_async(
Ok(crate::function::RefreshColumnResult {
rows_assigned: execution.result.rows_filled,
rows_failed: 0,
rows_remaining: 0,
rows_remaining: execution.rows_deferred,
source_version: execution.source_version,
published_version: (execution.result.rows_filled > 0)
.then_some(execution.result.version),
published_version: execution.published_version,
})
})))
}
@@ -414,6 +578,181 @@ mod tests {
table.add(batch).execute().await.unwrap();
}
/// The gate's reproducer: `b = coalesce(a, 0)` refreshed before `a`
/// must not bake zeros from `a`'s placeholder null.
#[tokio::test]
async fn test_dependent_refresh_cannot_fill_from_placeholder_null() {
let table = table_with("dependent_refresh_order", vec![1, 2, 3]).await;
table
.add_columns()
.computed("a", "x + 1")
.computed("b", "coalesce(a, 0)")
.execute()
.await
.unwrap();
let result = table.refresh_column("b").await.unwrap();
assert_eq!(result.rows_filled, 3);
assert_eq!(read(&table, "a").await, vec![Some(2), Some(3), Some(4)]);
assert_eq!(
table.count_rows(Some("b = a".to_string())).await.unwrap(),
3
);
assert_eq!(table.refresh_column("a").await.unwrap().rows_filled, 0);
// Appended rows: the input is filled in the new fragment first too.
append(&table, vec![10]).await;
assert_eq!(table.refresh_column("b").await.unwrap().rows_filled, 1);
assert_eq!(
table.count_rows(Some("b = 0".to_string())).await.unwrap(),
0
);
}
/// The gate's reproducer: a fragment appended between the input fill and
/// the requested column's fill has inputs the pass never covered, so it
/// is left unfilled rather than read as null.
#[tokio::test]
async fn test_dependent_refresh_fences_an_append_after_input_fill() {
let table = table_with("dependent_refresh_append_gap", vec![1, 2, 3]).await;
table
.add_columns()
.computed("a_gap", "x + 1")
.computed("b_gap", "coalesce(a_gap, 0)")
.execute()
.await
.unwrap();
let reached = Arc::new(tokio::sync::Notify::new());
let resume = Arc::new(tokio::sync::Notify::new());
super::PAUSES_BEFORE_FILL.lock().unwrap().push((
"b_gap".to_string(),
reached.clone(),
resume.clone(),
));
let refresh = {
let table = table.clone();
tokio::spawn(async move {
table
.refresh_column_async("b_gap")
.await
.unwrap()
.wait()
.await
})
};
reached.notified().await;
append(&table, vec![10]).await;
resume.notify_one();
let result = refresh.await.unwrap().unwrap();
assert_eq!(result.rows_assigned, 3);
// The target commit rebased over the append: the published version
// holds the deferred row, and the count is taken there.
assert_eq!(result.rows_remaining, 1);
table.refresh_column("a_gap").await.unwrap();
assert_eq!(table.count_rows(Some("b_gap = 0".into())).await.unwrap(), 0);
assert_eq!(table.refresh_column("b_gap").await.unwrap().rows_filled, 1);
}
/// The gate's reproducer: an append landing between two input fills is
/// rebased into the second's snapshot, but the first never covered it.
#[tokio::test]
async fn test_dependent_refresh_fences_an_append_between_input_fills() {
let table = table_with("dependent_refresh_between_inputs", vec![1, 2, 3]).await;
table
.add_columns()
.computed("a_mid", "x + 1")
.computed("c_mid", "x + 2")
.computed("b_mid", "coalesce(a_mid, 0) + coalesce(c_mid, 0)")
.execute()
.await
.unwrap();
let reached = Arc::new(tokio::sync::Notify::new());
let resume = Arc::new(tokio::sync::Notify::new());
super::PAUSES_BEFORE_FILL.lock().unwrap().push((
"c_mid".to_string(),
reached.clone(),
resume.clone(),
));
let refresh = {
let table = table.clone();
tokio::spawn(async move {
table
.refresh_column_async("b_mid")
.await
.unwrap()
.wait()
.await
})
};
reached.notified().await;
append(&table, vec![10]).await;
resume.notify_one();
let result = refresh.await.unwrap().unwrap();
assert_eq!(result.rows_assigned, 3);
// The appended row is in the reported source version but was deferred.
assert_eq!(result.rows_remaining, 1);
table.refresh_column("a_mid").await.unwrap();
table.refresh_column("c_mid").await.unwrap();
assert_eq!(table.count_rows(Some("b_mid = 0".into())).await.unwrap(), 0);
assert_eq!(table.refresh_column("b_mid").await.unwrap().rows_filled, 1);
assert_eq!(
table.count_rows(Some("b_mid = 23".into())).await.unwrap(),
1
);
}
/// A dependency commit is a publication even when the requested column
/// itself fills nothing.
#[tokio::test]
async fn test_refresh_async_reports_a_dependency_publication() {
let table = table_with("refresh_async_dependency_publication", vec![1, 2, 3]).await;
table
.add_columns()
.computed("a", "x + 1")
.computed("b", "nullif(a, a)")
.execute()
.await
.unwrap();
let result = table
.refresh_column_async("b")
.await
.unwrap()
.wait()
.await
.unwrap();
assert_eq!(result.rows_assigned, 0);
assert_eq!(result.published_version, Some(result.source_version));
}
/// The gate's reproducer: a dense graph is walked once, not once per path.
#[tokio::test]
async fn test_dependency_refresh_visits_each_column_once() {
let table = table_with("dependency_refresh_deduplicates", vec![1]).await;
let mut declaration = table.add_columns();
for i in 0..=6 {
let expression = if i == 0 {
"x + 1".to_string()
} else {
(0..i)
.map(|j| format!("a{j}"))
.collect::<Vec<_>>()
.join(" + ")
};
declaration = declaration.computed(format!("a{i}"), expression);
}
declaration.execute().await.unwrap();
let order = super::dependency_order(table.schema().await.unwrap().as_ref(), "a6").unwrap();
assert_eq!(order, (0..6).map(|i| format!("a{i}")).collect::<Vec<_>>());
// One commit per column, the requested one included.
let before = table.version().await.unwrap();
table.refresh_column("a6").await.unwrap();
assert_eq!(table.version().await.unwrap(), before + 7);
}
#[tokio::test]
async fn test_refresh_fills_a_declared_column() {
let table = table_with("refresh_fills", vec![1, 2, 3]).await;
@@ -20,6 +20,25 @@ fn job_result(name: &str) -> Value {
serde_json::from_str::<Value>(&fixture(name)).expect("remote Job fixture")["result"].clone()
}
fn assert_no_secret_values(value: &Value) {
match value {
Value::Object(values) => {
for (key, value) in values {
assert!(
!matches!(
key.as_str(),
"secret_value" | "secret_values" | "resolved_secret" | "resolved_secrets"
),
"client canonical value must not model resolved secret material"
);
assert_no_secret_values(value);
}
}
Value::Array(values) => values.iter().for_each(assert_no_secret_values),
_ => {}
}
}
#[test]
fn function_version_job_result_matches_shared_canonical_golden() {
let result = job_result("remote_function_job.json");
@@ -28,6 +47,7 @@ fn function_version_job_result_matches_shared_canonical_golden() {
assert_eq!(version.name(), "embed");
assert_eq!(version.version(), "fv_01K3EXACT");
assert_eq!(version.runtime_digest(), "sha256:runtime");
assert_eq!(version.required_secrets(), &["HF_TOKEN"]);
assert_eq!(
version.to_canonical_json().expect("canonical JSON"),
fixture("remote_function_version.canonical.json").trim()
@@ -142,3 +162,21 @@ fn floating_point_application_literals_are_rejected_consistently() {
.contains("floating-point Function literals")
);
}
#[test]
fn canonical_client_values_contain_secret_names_only() {
let result = job_result("remote_function_job.json");
let version = FunctionVersion::from_json(&result.to_string()).expect("FunctionVersion result");
let canonical: Value = serde_json::from_str(
&version
.to_canonical_json()
.expect("canonical FunctionVersion"),
)
.expect("canonical JSON");
assert_eq!(
canonical["required_secrets"],
serde_json::json!(["HF_TOKEN"])
);
assert_no_secret_values(&canonical);
}
@@ -6,6 +6,7 @@ use std::path::PathBuf;
use lancedb::Error;
use lancedb::function::FunctionRegistrationRequest;
use serde_json::Value;
fn fixture(name: &str) -> String {
let path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
@@ -14,6 +15,25 @@ fn fixture(name: &str) -> String {
fs::read_to_string(path).expect("fixture must be readable")
}
fn assert_no_secret_values(value: &Value) {
match value {
Value::Object(values) => {
for (key, value) in values {
assert!(
!matches!(
key.as_str(),
"secret_value" | "secret_values" | "resolved_secret" | "resolved_secrets"
),
"registration requests must not model resolved secret material"
);
assert_no_secret_values(value);
}
}
Value::Array(values) => values.iter().for_each(assert_no_secret_values),
_ => {}
}
}
#[test]
fn registration_request_matches_shared_canonical_golden() {
let request = FunctionRegistrationRequest::from_json(&fixture(
@@ -22,10 +42,33 @@ fn registration_request_matches_shared_canonical_golden() {
.expect("registration request");
assert_eq!(request.name, "normalize_score");
assert_eq!(request.artifact.adapter.kind, "scalar_to_arrow_batch");
assert_eq!(request.required_secrets, ["API_TOKEN"]);
assert!(request.secret_values.is_empty());
assert_eq!(
request.to_canonical_json().expect("canonical request"),
fixture("remote_function_registration_request.canonical.json").trim()
);
let value: Value =
serde_json::from_str(&request.to_canonical_json().expect("canonical request"))
.expect("request JSON");
assert_no_secret_values(&value);
}
#[test]
fn registration_request_serializes_secret_values_but_redacts_debug_output() {
let mut value: Value =
serde_json::from_str(&fixture("remote_function_registration_request.json")).unwrap();
value["secret_values"] = serde_json::json!({"API_TOKEN": "secret-plaintext"});
let request = FunctionRegistrationRequest::from_json(&value.to_string()).unwrap();
assert_eq!(request.secret_values["API_TOKEN"], "secret-plaintext");
let canonical = request.to_canonical_json().unwrap();
assert!(canonical.contains("secret-plaintext"));
let debug = format!("{request:?}");
assert!(debug.contains("API_TOKEN"));
assert!(debug.contains("[REDACTED]"));
assert!(!debug.contains("secret-plaintext"));
}
#[tokio::test]
@@ -24,6 +24,7 @@
},
"runtime_digest": "sha256:runtime",
"environment_digest": "sha256:environment",
"required_secrets": ["HF_TOKEN"],
"created_at": "2026-08-21T00:00:00Z"
},
"future_job": {"trace_id": "trace-1"}
@@ -1 +1 @@
{"artifact":{"adapter":{"kind":"scalar_to_arrow_batch","version":1},"content":{"data":"ZnJvbSBfX2Z1dHVyZV9fIGltcG9ydCBhbm5vdGF0aW9ucwoKZGVmIG5vcm1hbGl6ZV9zY29yZSh2YWx1ZTogZmxvYXQpIC0+IGZsb2F0OgogICAgcmV0dXJuIHZhbHVlIC8gMTAwLjAK","encoding":"base64"},"digest":"sha256:760784bdcef57b802f389b97804cc0b618aae39e86044733451bcd0b13089a7f","entrypoint":"normalize_score","kind":"python_callable"},"name":"normalize_score","runtime":{"env":{"MODE":"test"},"environment":{"kind":"pip","packages":["numpy>=2"]},"kind":"python","python_version":"3.12"},"signature":{"inputs":[{"arrow_type":"float64","name":"value","nullable":false}],"output":{"arrow_type":"float64","kind":"scalar","nullable":false}}}
{"artifact":{"adapter":{"kind":"scalar_to_arrow_batch","version":1},"content":{"data":"ZnJvbSBfX2Z1dHVyZV9fIGltcG9ydCBhbm5vdGF0aW9ucwoKZGVmIG5vcm1hbGl6ZV9zY29yZSh2YWx1ZTogZmxvYXQpIC0+IGZsb2F0OgogICAgcmV0dXJuIHZhbHVlIC8gMTAwLjAK","encoding":"base64"},"digest":"sha256:760784bdcef57b802f389b97804cc0b618aae39e86044733451bcd0b13089a7f","entrypoint":"normalize_score","kind":"python_callable"},"name":"normalize_score","required_secrets":["API_TOKEN"],"runtime":{"env":{"MODE":"test"},"environment":{"kind":"pip","packages":["numpy>=2"]},"kind":"python","python_version":"3.12"},"signature":{"inputs":[{"arrow_type":"float64","name":"value","nullable":false}],"output":{"arrow_type":"float64","kind":"scalar","nullable":false}}}
@@ -39,5 +39,8 @@
"env": {
"MODE": "test"
}
}
},
"required_secrets": [
"API_TOKEN"
]
}
@@ -1 +1 @@
{"artifact":{"digest":"sha256:code","entrypoint":"embed","kind":"python_callable"},"created_at":"2026-08-21T00:00:00Z","environment_digest":"sha256:environment","name":"embed","runtime":{"env":{"TOKENIZERS_PARALLELISM":"false"},"environment":{"kind":"pip","packages":["sentence-transformers>=3"]},"kind":"python","python_version":"3.12"},"runtime_digest":"sha256:runtime","signature":{"inputs":[{"arrow_type":"utf8","name":"text","nullable":true}],"output":{"arrow_type":"list<float32>","kind":"scalar","nullable":false}},"version":"fv_01K3EXACT"}
{"artifact":{"digest":"sha256:code","entrypoint":"embed","kind":"python_callable"},"created_at":"2026-08-21T00:00:00Z","environment_digest":"sha256:environment","name":"embed","required_secrets":["HF_TOKEN"],"runtime":{"env":{"TOKENIZERS_PARALLELISM":"false"},"environment":{"kind":"pip","packages":["sentence-transformers>=3"]},"kind":"python","python_version":"3.12"},"runtime_digest":"sha256:runtime","signature":{"inputs":[{"arrow_type":"utf8","name":"text","nullable":true}],"output":{"arrow_type":"list<float32>","kind":"scalar","nullable":false}},"version":"fv_01K3EXACT"}