mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-21 20:45:57 +00:00
Adds the client half of database-scoped named Secrets: a Secret is a
name and
an opaque value stored by the service, and a Function binds one to the
environment variable its library already reads. Secrets are addressed by
a
namespace path plus a name.
The UDF body is unchanged and stays portable — it reads `OPENAI_API_KEY`
the
way it always did, and the binding is what puts a value there:
```python
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
function = db.create_function(
analyze_caption,
secrets=[
EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
],
)
function.secret_bindings # the Secret's name, never its value
```
- `create_secret` / `alter_secret` / `list_secrets` / `describe_secret`
/
`drop_secret` on sync, async and remote connections, with the pyo3
binding
and the Rust client behind them. Each takes `namespace_path`
keyword-only,
defaulting to the root.
- **There is no read API, by construction rather than by policy** — no
code
path returns a stored credential, and `describe_secret` answers with
metadata only.
- `EnvVarSecret` is a pure local constructor: it contacts no server, so
it
cannot fail on a Secret that does not exist. It exists so that a bare
string
in that position — which would be a credential — is a `TypeError` rather
than a plausible-looking mistake that reads identically in a diff.
- `create_function(..., secrets=[...])` carries the bindings as
`secret_bindings`: a list of `SecretBinding` tagged by `kind`, so a
later
delivery mode is a variant rather than a sibling field. The value never
travels — it is resolved by the service when the Function runs, which is
what
lets a rotation reach columns already pinned to an older
FunctionVersion.
- A binding names its Secret as a `SecretReference` of `{name,
namespace_path}`
rather than one joined string, so no delimiter has to be excluded from
every
name and segment forever, and `ClientConfig.id_delimiter` cannot
contradict
an identity built on a fixed separator.
- A root namespace is omitted from the request body rather than sent
empty, so
a root request is byte-identical to one from a client that predates
namespaces. Tests pin it.
This is the client surface the design's §4 describes; the service side
lives in
sophon.
**Previously split across two PRs.** Namespace addressing was #4151,
stacked on
this one; it is folded in here so the Secret identity contract — name,
namespace path, and the binding that carries both — is reviewable as one
piece
rather than as a shape introduced and then replaced.
## Identifier safety, merged from #4189
**#4189 is merged into this branch**, so the client half of Secrets and
the
guards on the identity it puts in the URL are one PR. What it added:
- Components are checked where the identifier is built, before a request
is
constructed. `create_secret("../jobs", value)` no longer resolves to
`/v1/jobs/create` and delivers a credential-bearing body to a route with
none
of this one's body suppression.
- Each component is percent-encoded and joined by the delimiter, so
nothing
inside a component can end the path segment or add one.
- A component may not be empty, a relative segment (`.`, `..`, and their
`%2e`
spellings), or the delimiter itself — the three ways a component erases
a
boundary the split has to recover. `["prod", ""]` joined to `prod$`,
which
reads back as `["prod"]`.
- `$` is the only accepted `id_delimiter`, refused at client
construction.
`ClientConfig.id_delimiter` remains, since the identifier grammar comes
from
the Lance REST catalog standard, but a value that would produce
identifiers no
service splits the caller's way is now an error where it was written.
- One `build_object_identifier` and one character set serve tables,
namespaces,
Secrets, Functions and materialized views.
Components are checked for *addressability*, not a character set: the
name's
own grammar stays each object's own, so a catalog database keeps the `/`
that
`RemoteCatalog::validate_name` allows.
## Known shortcoming
`secret_bindings` is omitted from a registration body when empty, so a
client
that binds nothing sends what a client without bindings sends. When a
client
does bind a Secret and the service does not know the field, the field is
ignored: registration succeeds, the returned version carries no
bindings, and
the Function fails at execution with the variable unset, far from the
call that
asked for it.
`ServerVersion` is how this codebase refuses a feature the service is
too old
for, and it gates five features already. It does not gate this one: it
is held
per table, and registering a Function is a database-level call. Noted at
the
field in `remote/db.rs`; wiring the gate is follow-up work.
**Tests:** lancedb lib 1340 passed, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, plus Python tests across both slices.
Rebased onto `main` after #4176 (OCI Function identity), #4191 (`.`/`..`
table
names) and #4195 (remote catalogs).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
143 lines
4.9 KiB
TypeScript
143 lines
4.9 KiB
TypeScript
// SPDX-License-Identifier: Apache-2.0
|
|
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
|
|
|
|
import * as http from "http";
|
|
import { Catalog, connectCatalog } from "../lancedb";
|
|
|
|
type RecordedRequest = {
|
|
url: string;
|
|
headers: http.IncomingHttpHeaders;
|
|
body: Record<string, unknown>;
|
|
};
|
|
|
|
async function withCatalog(
|
|
responses: [number, unknown][],
|
|
callback: (catalog: Catalog, requests: RecordedRequest[]) => Promise<void>,
|
|
) {
|
|
const requests: RecordedRequest[] = [];
|
|
const server = http.createServer(async (req, res) => {
|
|
const chunks: Buffer[] = [];
|
|
for await (const chunk of req) chunks.push(Buffer.from(chunk));
|
|
const body = Buffer.concat(chunks).toString();
|
|
requests.push({
|
|
url: req.url ?? "",
|
|
headers: req.headers,
|
|
body: body ? JSON.parse(body) : {},
|
|
});
|
|
const [status, response] = responses.shift() ?? [
|
|
500,
|
|
{ error: "Unexpected request" },
|
|
];
|
|
res.writeHead(status, { "content-type": "application/json" });
|
|
res.end(status === 204 ? undefined : JSON.stringify(response));
|
|
});
|
|
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
|
|
const address = server.address();
|
|
if (!address || typeof address === "string")
|
|
throw new Error("Missing server address");
|
|
try {
|
|
const catalog = await connectCatalog(`http://127.0.0.1:${address.port}`, {
|
|
apiKey: "secret",
|
|
sqlHostOverride: "grpc+tls://sql.example.com:10026",
|
|
clientConfig: {
|
|
extraHeaders: {
|
|
"X-LanceDB-Database": "wrong-static",
|
|
"X-LanceDB-Database-Prefix": "wrong",
|
|
},
|
|
},
|
|
headerProvider: () => ({
|
|
"X-LanceDB-Database": "wrong-dynamic",
|
|
"X-LanceDB-Database-Prefix": "wrong",
|
|
authorization: "Bearer refreshed",
|
|
}),
|
|
});
|
|
await callback(catalog, requests);
|
|
expect(responses).toHaveLength(0);
|
|
} finally {
|
|
server.closeAllConnections();
|
|
await new Promise<void>((resolve) => server.close(() => resolve()));
|
|
}
|
|
}
|
|
|
|
describe("remote catalog", () => {
|
|
it("uses root namespace routes and preserves independent database scope", async () => {
|
|
await withCatalog(
|
|
[
|
|
[204, null],
|
|
[200, { tables: [] }],
|
|
[200, {}],
|
|
[200, { tables: [] }],
|
|
[200, { tables: [] }],
|
|
// biome-ignore lint/style/useNamingConvention: server wire format
|
|
[200, { namespaces: ["team/search"], page_token: "next" }],
|
|
[204, null],
|
|
],
|
|
async (catalog, requests) => {
|
|
const first = await catalog.createDatabase("team/search", {
|
|
existOk: true,
|
|
});
|
|
expect(await first.tableNames()).toEqual([]);
|
|
const second = await catalog.connectDatabase("other");
|
|
expect(await second.tableNames()).toEqual([]);
|
|
expect(await first.tableNames()).toEqual([]);
|
|
expect(
|
|
await catalog.listDatabases({ limit: 1, pageToken: "a/b" }),
|
|
).toEqual({ databases: ["team/search"], pageToken: "next" });
|
|
await catalog.dropDatabase("team/search", { ignoreMissing: true });
|
|
expect(requests[0].url).toBe("/v1/namespace/team%2Fsearch/create");
|
|
expect(requests[0].body).toEqual({ mode: "ExistOk" });
|
|
expect(requests[5].url).toBe(
|
|
"/v1/namespace/$/list?limit=1&page_token=a%2Fb",
|
|
);
|
|
expect(requests[6].body).toEqual({
|
|
mode: "Skip",
|
|
behavior: "Restrict",
|
|
});
|
|
for (const [i, request] of requests.entries()) {
|
|
expect(request.headers["x-lancedb-database"]).toBe(
|
|
i === 1 || i === 4 ? "team/search" : i === 3 ? "other" : undefined,
|
|
);
|
|
expect(request.headers["x-lancedb-database-prefix"]).toBeUndefined();
|
|
expect(request.headers.authorization).toBe("Bearer refreshed");
|
|
}
|
|
},
|
|
);
|
|
});
|
|
|
|
it("propagates lifecycle errors and sends restricted drops", async () => {
|
|
await withCatalog(
|
|
[
|
|
[404, {}],
|
|
[409, {}],
|
|
[400, {}],
|
|
[404, {}],
|
|
],
|
|
async (catalog, requests) => {
|
|
await expect(catalog.connectDatabase("missing")).rejects.toThrow(
|
|
"missing",
|
|
);
|
|
await expect(catalog.createDatabase("exists")).rejects.toThrow(
|
|
"exists",
|
|
);
|
|
await expect(catalog.dropDatabase("full")).rejects.toThrow();
|
|
await catalog.dropDatabase("missing", { ignoreMissing: true });
|
|
expect(requests[2].body).toEqual({
|
|
mode: "Fail",
|
|
behavior: "Restrict",
|
|
});
|
|
},
|
|
);
|
|
});
|
|
|
|
it("validates endpoints and pagination", async () => {
|
|
await expect(connectCatalog("/tmp/catalog")).rejects.toThrow();
|
|
const catalog = await connectCatalog("http://127.0.0.1:1");
|
|
for (const limit of [0, -1, 1.5, 2147483648]) {
|
|
await expect(catalog.listDatabases({ limit })).rejects.toThrow("limit");
|
|
}
|
|
await expect(catalog.connectDatabase("a$b")).rejects.toThrow(
|
|
"Invalid database name",
|
|
);
|
|
});
|
|
});
|