feat(secrets): named Secrets, bindings, and namespace addressing (#4150)

Adds the client half of database-scoped named Secrets: a Secret is a
name and
an opaque value stored by the service, and a Function binds one to the
environment variable its library already reads. Secrets are addressed by
a
namespace path plus a name.

The UDF body is unchanged and stays portable — it reads `OPENAI_API_KEY`
the
way it always did, and the binding is what puts a value there:

```python
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
function = db.create_function(
    analyze_caption,
    secrets=[
        EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
    ],
)
function.secret_bindings   # the Secret's name, never its value
```

- `create_secret` / `alter_secret` / `list_secrets` / `describe_secret`
/
`drop_secret` on sync, async and remote connections, with the pyo3
binding
and the Rust client behind them. Each takes `namespace_path`
keyword-only,
  defaulting to the root.
- **There is no read API, by construction rather than by policy** — no
code
  path returns a stored credential, and `describe_secret` answers with
  metadata only.
- `EnvVarSecret` is a pure local constructor: it contacts no server, so
it
cannot fail on a Secret that does not exist. It exists so that a bare
string
in that position — which would be a credential — is a `TypeError` rather
  than a plausible-looking mistake that reads identically in a diff.
- `create_function(..., secrets=[...])` carries the bindings as
`secret_bindings`: a list of `SecretBinding` tagged by `kind`, so a
later
delivery mode is a variant rather than a sibling field. The value never
travels — it is resolved by the service when the Function runs, which is
what
lets a rotation reach columns already pinned to an older
FunctionVersion.
- A binding names its Secret as a `SecretReference` of `{name,
namespace_path}`
rather than one joined string, so no delimiter has to be excluded from
every
name and segment forever, and `ClientConfig.id_delimiter` cannot
contradict
  an identity built on a fixed separator.
- A root namespace is omitted from the request body rather than sent
empty, so
  a root request is byte-identical to one from a client that predates
  namespaces. Tests pin it.

This is the client surface the design's §4 describes; the service side
lives in
sophon.

**Previously split across two PRs.** Namespace addressing was #4151,
stacked on
this one; it is folded in here so the Secret identity contract — name,
namespace path, and the binding that carries both — is reviewable as one
piece
rather than as a shape introduced and then replaced.

## Identifier safety, merged from #4189

**#4189 is merged into this branch**, so the client half of Secrets and
the
guards on the identity it puts in the URL are one PR. What it added:

- Components are checked where the identifier is built, before a request
is
  constructed. `create_secret("../jobs", value)` no longer resolves to
`/v1/jobs/create` and delivers a credential-bearing body to a route with
none
  of this one's body suppression.
- Each component is percent-encoded and joined by the delimiter, so
nothing
  inside a component can end the path segment or add one.
- A component may not be empty, a relative segment (`.`, `..`, and their
`%2e`
spellings), or the delimiter itself — the three ways a component erases
a
boundary the split has to recover. `["prod", ""]` joined to `prod$`,
which
  reads back as `["prod"]`.
- `$` is the only accepted `id_delimiter`, refused at client
construction.
`ClientConfig.id_delimiter` remains, since the identifier grammar comes
from
the Lance REST catalog standard, but a value that would produce
identifiers no
  service splits the caller's way is now an error where it was written.
- One `build_object_identifier` and one character set serve tables,
namespaces,
  Secrets, Functions and materialized views.

Components are checked for *addressability*, not a character set: the
name's
own grammar stays each object's own, so a catalog database keeps the `/`
that
`RemoteCatalog::validate_name` allows.

## Known shortcoming

`secret_bindings` is omitted from a registration body when empty, so a
client
that binds nothing sends what a client without bindings sends. When a
client
does bind a Secret and the service does not know the field, the field is
ignored: registration succeeds, the returned version carries no
bindings, and
the Function fails at execution with the variable unset, far from the
call that
asked for it.

`ServerVersion` is how this codebase refuses a feature the service is
too old
for, and it gates five features already. It does not gate this one: it
is held
per table, and registering a Function is a database-level call. Noted at
the
field in `remote/db.rs`; wiring the gate is follow-up work.

**Tests:** lancedb lib 1340 passed, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, plus Python tests across both slices.
Rebased onto `main` after #4176 (OCI Function identity), #4191 (`.`/`..`
table
names) and #4195 (remote catalogs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jonathan Hsieh
2026-09-16 17:00:23 -07:00
committed by GitHub
co-authored by Claude Opus 5
parent f8d73b3447
commit 60a1b4c219
30 changed files with 2671 additions and 190 deletions
+1 -1
View File
@@ -91,7 +91,7 @@ def test_catalog_sync_scope_and_serialization(catalog_server):
assert requests[0][0] == "/v1/namespace/team%2Fsearch/create"
assert requests[0][2] == {"mode": "ExistOk"}
assert requests[1][0] == "/v1/namespace/team%2Fsearch/describe"
assert requests[4][0] == "/v1/namespace/%24/list?limit=1&page_token=a%2Fb"
assert requests[4][0] == "/v1/namespace/$/list?limit=1&page_token=a%2Fb"
assert requests[5][2] == {"mode": "Skip", "behavior": "Restrict"}
for i, (_, headers, _) in enumerate(requests):
headers = {key.lower(): value for key, value in headers.items()}
@@ -13,7 +13,9 @@ from lancedb.functions import (
FunctionBinding,
FunctionVersion,
PythonRuntimeSpec,
SecretBinding,
RefreshColumnResult,
SecretReference,
)
from lancedb.table import AsyncTable
@@ -37,6 +39,22 @@ def job_result(name: str) -> dict:
return json.loads(fixture(name))["result"]
def assert_no_secret_values(value):
"""No client value models a resolved credential, at any nesting depth."""
if isinstance(value, dict):
for key, child in value.items():
assert key not in {
"secret_value",
"secret_values",
"resolved_secret",
"resolved_secrets",
}
assert_no_secret_values(child)
elif isinstance(value, list):
for child in value:
assert_no_secret_values(child)
def test_public_function_values_are_in_api_reference():
docs = Path(__file__).parents[3] / "docs" / "src" / "python" / "python.md"
rendered = docs.read_text()
@@ -96,6 +114,11 @@ def test_function_version_identity_is_immutable_and_exact():
assert version.version == "1"
assert version.image.manifest_digest.startswith("sha256:")
assert version.version != version.image.manifest_digest
assert list(version.secret_bindings) == [
SecretBinding(
kind="env", variable="HF_TOKEN", secret_ref=SecretReference(name="hf-prod")
)
]
with pytest.raises((TypeError, ValueError)):
version.version = "1"
@@ -282,6 +305,31 @@ def test_refresh_result_rejects_non_u64_values(field):
RefreshColumnResult.from_json(json.dumps(value))
def test_canonical_client_values_carry_bindings_and_no_credentials():
"""A binding names a Secret; the credential behind it has no client field."""
version = FunctionVersion.from_json(
json.dumps(job_result("remote_function_job.json"))
)
canonical = json.loads(version.to_canonical_json())
assert canonical["secret_bindings"] == [
{"kind": "env", "variable": "HF_TOKEN", "secret_ref": {"name": "hf-prod"}}
]
assert_no_secret_values(canonical)
def test_a_version_without_bindings_omits_the_field_in_both_directions():
"""A Function that binds nothing carries no ``secret_bindings`` key.
Absent decodes as an empty list, and an empty list serializes back to
absent.
"""
value = job_result("remote_function_job.json")
del value["secret_bindings"]
version = FunctionVersion.from_json(json.dumps(value))
assert list(version.secret_bindings) == []
assert "secret_bindings" not in json.loads(version.to_canonical_json())
class _FunctionDeclarationInner:
def __init__(self):
self.calls = []
@@ -11,6 +11,7 @@ import types
from datetime import date
import http.server
import json
import os
from pathlib import Path
import subprocess
import sys
@@ -24,11 +25,14 @@ import pytest
import lancedb
from lancedb.functions import (
PythonRuntimeSpec,
SecretBinding,
SecretReference,
UdfDefinition,
_canonical_arrow_type,
_GRAMMAR_PRIMITIVES,
udf,
)
from lancedb.secrets import EnvVarSecret
THRESHOLD = 20
_CACHE = None
@@ -59,6 +63,15 @@ def normalize_score(value: float) -> float:
return value / 100.0
@udf(
pip=["openai==3.7.0"],
env={"MODE": "test"},
python_version="3.12",
)
def analyze_caption(caption: str) -> str:
return caption.strip()
def test_scalar_udf_matches_shared_registration_golden_and_remains_callable():
assert isinstance(normalize_score, UdfDefinition)
assert normalize_score(25.0) == 0.25
@@ -75,6 +88,286 @@ def test_scalar_udf_matches_shared_registration_golden_and_remains_callable():
}
def test_secret_bound_udf_matches_its_shared_registration_golden():
assert analyze_caption(" hello ") == "hello"
bound = analyze_caption.bind_secrets(
[EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")]
)
assert (
bound.to_canonical_json()
== (FIXTURES / "remote_function_secret_registration_request.canonical.json")
.read_text()
.strip()
)
def test_a_namespaced_binding_records_the_path_and_the_name():
"""A binding names the parts, so nothing has to be parsed back out.
A root binding carries no path at all: the field is absent rather than an
empty list, so a binding states a namespace only when it has one.
"""
root = EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
assert root.secret_namespace_path == []
nested = EnvVarSecret(
secret_name="openai-prod",
env_variable="OPENAI_API_KEY",
secret_namespace_path=["prod", "vision"],
)
assert nested.secret_namespace_path == ["prod", "vision"]
assert nested != root
bound = analyze_caption.bind_secrets([nested])
assert list(bound.secret_bindings) == [
SecretBinding(
kind="env",
variable="OPENAI_API_KEY",
secret_ref=SecretReference(
name="openai-prod", namespace_path=("prod", "vision")
),
)
]
at_root = analyze_caption.bind_secrets([root])
assert list(at_root.secret_bindings) == [
SecretBinding(
kind="env",
variable="OPENAI_API_KEY",
secret_ref=SecretReference(name="openai-prod"),
)
]
# A root binding carries no path at all on the wire.
canonical = json.loads(at_root.to_canonical_json())
assert canonical["secret_bindings"] == [
{
"kind": "env",
"variable": "OPENAI_API_KEY",
"secret_ref": {"name": "openai-prod"},
}
]
def test_a_namespace_path_is_validated_locally():
# The charset is the service's, not a delimiter's: a reference is never
# joined, so a segment cannot make anything parse two ways.
with pytest.raises(ValueError):
EnvVarSecret(
secret_name="openai-prod",
env_variable="K",
secret_namespace_path=["with$delim"],
)
with pytest.raises(ValueError):
EnvVarSecret(
secret_name="openai-prod", env_variable="K", secret_namespace_path=["a/b"]
)
# A bare string is a plausible mistake with the wrong meaning.
with pytest.raises(TypeError):
EnvVarSecret(
secret_name="openai-prod", env_variable="K", secret_namespace_path="prod"
)
def test_an_unbound_request_carries_no_binding_at_all():
"""Binding is a registration-time decision, so the definition holds none.
The decorator declares nothing about secrets, which is what makes the PRD's
claim true: a Function's source and its registration request are identical
whether or not a credential is later bound to it.
"""
unbound = json.loads(analyze_caption.registration_request.to_canonical_json())
assert "secret_bindings" not in unbound
assert "OPENAI_API_KEY" not in json.dumps(unbound)
def test_binding_a_secret_leaves_the_packaged_artifact_untouched():
"""The artifact is source bytes and nothing else, with or without secrets."""
bound = analyze_caption.bind_secrets(
[EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")]
)
assert bound.artifact == analyze_caption.registration_request.artifact
assert bound.artifact.digest == analyze_caption.registration_request.artifact.digest
def test_a_function_declaring_no_secret_is_registered_exactly_as_before():
"""The compatibility claim: nothing about the no-secret path moves."""
assert (
normalize_score.bind_secrets(None).to_canonical_json()
== normalize_score.registration_request.to_canonical_json()
)
assert (
"secret_bindings"
not in normalize_score.registration_request.to_canonical_json()
)
def test_a_function_binds_each_variable_once():
with pytest.raises(ValueError, match="binds each environment variable once"):
analyze_caption.bind_secrets(
[
EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY"),
EnvVarSecret(
secret_name="openai-staging", env_variable="OPENAI_API_KEY"
),
]
)
def test_bindings_may_not_collide_with_plain_configuration():
"""`env` is stored with the Function; a Secret is not. Refuse, do not pick."""
with pytest.raises(ValueError, match="must be disjoint"):
analyze_caption.bind_secrets(
[EnvVarSecret(secret_name="mode-prod", env_variable="MODE")]
)
def test_a_binding_envelope_reaches_the_service_for_it_to_judge():
"""Binding rules are the service's: it owns the runtime the names land in.
The client sends what it was given, so a rule it duplicated could disagree
with the service's without either side noticing. What is checked here is
that the envelope arrives intact -- the shape the service judges is the
shape the caller wrote.
"""
with _mock_remote_function_catalog() as (host, state):
db = lancedb.connect(
"db://dev",
api_key="fake",
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
bindings = [
EnvVarSecret(secret_name=f"secret-{index}", env_variable=f"TOKEN_{index}")
for index in range(17)
]
db.create_function(normalize_score, secrets=bindings)
sent = state["requests"][0][1]
assert len(sent["secret_bindings"]) == 17
assert {
"kind": "env",
"variable": "TOKEN_0",
"secret_ref": {"name": "secret-0"},
} in sent["secret_bindings"]
_SECRET_DEBUG_LOG_SOURCE = """
import http.server
import json
import threading
import lancedb
class Handler(http.server.BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def do_POST(self):
self.rfile.read(int(self.headers.get("Content-Length", "0")))
payload = json.dumps({}).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
server = http.server.ThreadingHTTPServer(("localhost", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
try:
db = lancedb.connect(
"db://dev",
api_key="API_KEY_SENTINEL",
host_override="http://localhost:%d" % server.server_address[1],
client_config={"retry_config": {"retries": 0}},
)
db.create_secret("openai-prod", "SECRET_VALUE_SENTINEL")
finally:
server.shutdown()
"""
def test_a_credential_never_reaches_a_debug_log(tmp_path):
"""The logger sees the serialized body, so no value-side redaction reaches it.
Runs in a subprocess because the Rust logger reads ``LANCEDB_LOG`` once, at
import.
"""
script = tmp_path / "write_secret.py"
script.write_text(_SECRET_DEBUG_LOG_SOURCE)
result = subprocess.run(
[sys.executable, str(script)],
check=True,
capture_output=True,
text=True,
env={**os.environ, "LANCEDB_LOG": "debug"},
)
output = result.stdout + result.stderr
# Without this the test passes when debug logging is simply off.
assert "Sending request_id=" in output, output
assert "SECRET_VALUE_SENTINEL" not in output
assert "API_KEY_SENTINEL" not in output
def test_a_credential_value_is_rejected_in_the_binding_position():
"""The one mistake the typed binding exists to stop."""
with pytest.raises(TypeError, match="EnvVarSecret"):
analyze_caption.bind_secrets(["sk-live-0001"])
@pytest.mark.parametrize(
("secret", "variable", "message"),
[
("openai-prod", "not-a-var", "invalid environment variable name"),
("openai-prod", "API-TOKEN", "invalid environment variable name"),
("not a name", "API_TOKEN", "invalid Secret name"),
("openai$prod", "API_TOKEN", "invalid Secret name"),
],
)
def test_a_binding_validates_both_names_locally(secret, variable, message):
with pytest.raises(ValueError, match=message):
EnvVarSecret(secret_name=secret, env_variable=variable)
def test_a_secret_name_admits_what_a_namespace_name_does():
"""A Secret has to be nameable wherever a namespace already is.
LanceDB namespace and table names are `[A-Za-z0-9_.-]` with no rule about
which character comes first, so a name may lead with `_`, `-` or `.`.
Anything narrower here would leave Secrets unaddressable inside namespaces
that already exist -- the reason periods are admitted is the reason the
edges are too.
"""
for name in ["openai.prod.v1", ".hidden", "_internal", "-lead", "trailing."]:
binding = EnvVarSecret(secret_name=name, env_variable="OPENAI_API_KEY")
assert binding.secret_name == name
for name in ["", "with/slash", "with$delimiter", "a" * 256]:
with pytest.raises(ValueError, match="invalid Secret name"):
EnvVarSecret(secret_name=name, env_variable="OPENAI_API_KEY")
# A namespace segment follows the same rule, and LanceDB already admits
# these shapes as namespace names -- so a Secret is addressable inside one.
for segment in [".hidden", "_internal", "-lead", "trailing."]:
binding = EnvVarSecret(
secret_name="openai-prod",
env_variable="OPENAI_API_KEY",
secret_namespace_path=[segment],
)
assert binding.secret_namespace_path == [segment]
for segment in ["", "with/slash", "with$delimiter"]:
with pytest.raises(ValueError, match="invalid namespace path segment"):
EnvVarSecret(
secret_name="openai-prod",
env_variable="OPENAI_API_KEY",
secret_namespace_path=[segment],
)
def _main_udf_source(
*, threshold: int = 20, input_annotation: str = "int", comparison: str = ">="
) -> str:
@@ -1232,7 +1525,20 @@ def _mock_remote_function_catalog():
body = json.loads(self.rfile.read(length) or b"{}")
state["requests"].append((self.path, body))
status = 200
if self.path == "/v1/function/normalize_score/create":
# `{id}` is the Function name, so match on the shape rather than on
# one name: these tests register more than one Function.
parts = self.path.strip("/").split("/")
function_action = (
(urllib.parse.unquote(parts[2]), parts[3])
if len(parts) == 4 and parts[0] == "v1" and parts[1] == "function"
else (None, None)
)
secret_action = (
(urllib.parse.unquote(parts[2]), parts[3])
if len(parts) == 4 and parts[0] == "v1" and parts[1] == "secret"
else (None, None)
)
if function_action[1] == "create":
state["version"] = {
"name": "normalize_score",
"version": FUNCTION_VERSION,
@@ -1246,6 +1552,7 @@ def _mock_remote_function_catalog():
).read_text()
)["image"],
"signature": body["signature"],
"secret_bindings": body.get("secret_bindings", []),
"created_at": "2026-08-21T00:00:00Z",
}
response = {"job_id": "job-register"}
@@ -1264,6 +1571,15 @@ def _mock_remote_function_catalog():
elif self.path == "/v1/function/normalize_score/drop":
assert body == {"version": FUNCTION_VERSION}
response = {"dropped": True}
elif secret_action[1] in ("create", "alter"):
# The Secret is the path identifier, so the body is the value.
assert set(body) == {"value"}
assert secret_action[0] == "openai-prod"
response = {}
elif secret_action[1] == "drop":
assert secret_action[0] == "openai-prod"
assert body == {}
response = {}
else:
status = 404
response = {"error": "not found"}
@@ -1276,6 +1592,16 @@ def _mock_remote_function_catalog():
for key, values in urllib.parse.parse_qs(url.query).items()
}
state["requests"].append((url.path, query))
if url.path == "/v1/namespace/$/secret/list":
if "page_token" not in query:
self._write_response(
200,
{"secrets": [{"name": "openai-prod"}], "page_token": "next"},
)
else:
assert query["page_token"] == "next"
self._write_response(200, {"secrets": [{"name": "hf-prod"}]})
return
if url.path != "/v1/namespace/$/function/list":
self._write_response(404, {"error": "not found"})
return
@@ -1330,6 +1656,92 @@ def test_remote_registration_job_and_exact_version_reopen_round_trip():
assert create_request == expected_request
def test_remote_registration_sends_bindings_and_never_a_credential():
with _mock_remote_function_catalog() as (host, state):
db = lancedb.connect(
"db://dev",
api_key="fake",
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
created = db.create_function(
analyze_caption,
secrets=[
EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
],
)
assert list(created.secret_bindings) == [
SecretBinding(
kind="env",
variable="OPENAI_API_KEY",
secret_ref=SecretReference(name="openai-prod"),
)
]
path, create_request = state["requests"][0]
assert path == "/v1/function/analyze_caption/create"
assert create_request["secret_bindings"] == [
{
"kind": "env",
"variable": "OPENAI_API_KEY",
"secret_ref": {"name": "openai-prod"},
}
]
# The request names a Secret and carries nothing that could be one. The
# Function's own name is the path identifier rather than a body field, so
# it is the one key the body does not repeat.
expected = json.loads(
analyze_caption.bind_secrets(
[EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")]
).to_canonical_json()
)
assert expected.pop("name") == "analyze_caption"
assert create_request == expected
def test_remote_secret_verbs_round_trip():
with _mock_remote_function_catalog() as (host, state):
db = lancedb.connect(
"db://dev",
api_key="fake",
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
assert db.create_secret("openai-prod", "sk-live-0001") is None
assert db.alter_secret("openai-prod", "sk-live-0002") is None
assert db.list_secrets() == ["openai-prod", "hf-prod"]
assert db.drop_secret("openai-prod") is None
routes = [path for path, _ in state["requests"]]
assert routes == [
"/v1/secret/openai-prod/create",
"/v1/secret/openai-prod/alter",
"/v1/namespace/$/secret/list",
"/v1/namespace/$/secret/list",
"/v1/secret/openai-prod/drop",
]
# The Secret is the path identifier, so the body is the value alone.
assert state["requests"][0][1] == {"value": "sk-live-0001"}
# Listing is a GET: the first page asks for nothing, the second resumes on
# the token the server handed back, and neither carries a body.
assert state["requests"][2][1] == {}
assert state["requests"][3][1] == {"page_token": "next"}
def test_building_a_binding_contacts_no_server():
"""A binding is a local value: it says nothing about whether the Secret exists.
Existence is the server's answer at registration, where a mistyped name is a
clear error rather than a client-side check that was already stale.
"""
with _mock_remote_function_catalog() as (_host, state):
binding = EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
assert binding.secret_name == "openai-prod"
assert binding.env_variable == "OPENAI_API_KEY"
assert state["requests"] == []
def test_blocking_remote_registration_returns_function_version():
with _mock_remote_function_catalog() as (host, state):
db = lancedb.connect(