Files
lancedb/nodejs/__test__/catalog.test.ts
T
Jonathan HsiehandClaude Opus 5 60a1b4c219 feat(secrets): named Secrets, bindings, and namespace addressing (#4150)
Adds the client half of database-scoped named Secrets: a Secret is a
name and
an opaque value stored by the service, and a Function binds one to the
environment variable its library already reads. Secrets are addressed by
a
namespace path plus a name.

The UDF body is unchanged and stays portable — it reads `OPENAI_API_KEY`
the
way it always did, and the binding is what puts a value there:

```python
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
function = db.create_function(
    analyze_caption,
    secrets=[
        EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
    ],
)
function.secret_bindings   # the Secret's name, never its value
```

- `create_secret` / `alter_secret` / `list_secrets` / `describe_secret`
/
`drop_secret` on sync, async and remote connections, with the pyo3
binding
and the Rust client behind them. Each takes `namespace_path`
keyword-only,
  defaulting to the root.
- **There is no read API, by construction rather than by policy** — no
code
  path returns a stored credential, and `describe_secret` answers with
  metadata only.
- `EnvVarSecret` is a pure local constructor: it contacts no server, so
it
cannot fail on a Secret that does not exist. It exists so that a bare
string
in that position — which would be a credential — is a `TypeError` rather
  than a plausible-looking mistake that reads identically in a diff.
- `create_function(..., secrets=[...])` carries the bindings as
`secret_bindings`: a list of `SecretBinding` tagged by `kind`, so a
later
delivery mode is a variant rather than a sibling field. The value never
travels — it is resolved by the service when the Function runs, which is
what
lets a rotation reach columns already pinned to an older
FunctionVersion.
- A binding names its Secret as a `SecretReference` of `{name,
namespace_path}`
rather than one joined string, so no delimiter has to be excluded from
every
name and segment forever, and `ClientConfig.id_delimiter` cannot
contradict
  an identity built on a fixed separator.
- A root namespace is omitted from the request body rather than sent
empty, so
  a root request is byte-identical to one from a client that predates
  namespaces. Tests pin it.

This is the client surface the design's §4 describes; the service side
lives in
sophon.

**Previously split across two PRs.** Namespace addressing was #4151,
stacked on
this one; it is folded in here so the Secret identity contract — name,
namespace path, and the binding that carries both — is reviewable as one
piece
rather than as a shape introduced and then replaced.

## Identifier safety, merged from #4189

**#4189 is merged into this branch**, so the client half of Secrets and
the
guards on the identity it puts in the URL are one PR. What it added:

- Components are checked where the identifier is built, before a request
is
  constructed. `create_secret("../jobs", value)` no longer resolves to
`/v1/jobs/create` and delivers a credential-bearing body to a route with
none
  of this one's body suppression.
- Each component is percent-encoded and joined by the delimiter, so
nothing
  inside a component can end the path segment or add one.
- A component may not be empty, a relative segment (`.`, `..`, and their
`%2e`
spellings), or the delimiter itself — the three ways a component erases
a
boundary the split has to recover. `["prod", ""]` joined to `prod$`,
which
  reads back as `["prod"]`.
- `$` is the only accepted `id_delimiter`, refused at client
construction.
`ClientConfig.id_delimiter` remains, since the identifier grammar comes
from
the Lance REST catalog standard, but a value that would produce
identifiers no
  service splits the caller's way is now an error where it was written.
- One `build_object_identifier` and one character set serve tables,
namespaces,
  Secrets, Functions and materialized views.

Components are checked for *addressability*, not a character set: the
name's
own grammar stays each object's own, so a catalog database keeps the `/`
that
`RemoteCatalog::validate_name` allows.

## Known shortcoming

`secret_bindings` is omitted from a registration body when empty, so a
client
that binds nothing sends what a client without bindings sends. When a
client
does bind a Secret and the service does not know the field, the field is
ignored: registration succeeds, the returned version carries no
bindings, and
the Function fails at execution with the variable unset, far from the
call that
asked for it.

`ServerVersion` is how this codebase refuses a feature the service is
too old
for, and it gates five features already. It does not gate this one: it
is held
per table, and registering a Function is a database-level call. Noted at
the
field in `remote/db.rs`; wiring the gate is follow-up work.

**Tests:** lancedb lib 1340 passed, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, plus Python tests across both slices.
Rebased onto `main` after #4176 (OCI Function identity), #4191 (`.`/`..`
table
names) and #4195 (remote catalogs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:00:23 -07:00

143 lines
4.9 KiB
TypeScript

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
import * as http from "http";
import { Catalog, connectCatalog } from "../lancedb";
type RecordedRequest = {
url: string;
headers: http.IncomingHttpHeaders;
body: Record<string, unknown>;
};
async function withCatalog(
responses: [number, unknown][],
callback: (catalog: Catalog, requests: RecordedRequest[]) => Promise<void>,
) {
const requests: RecordedRequest[] = [];
const server = http.createServer(async (req, res) => {
const chunks: Buffer[] = [];
for await (const chunk of req) chunks.push(Buffer.from(chunk));
const body = Buffer.concat(chunks).toString();
requests.push({
url: req.url ?? "",
headers: req.headers,
body: body ? JSON.parse(body) : {},
});
const [status, response] = responses.shift() ?? [
500,
{ error: "Unexpected request" },
];
res.writeHead(status, { "content-type": "application/json" });
res.end(status === 204 ? undefined : JSON.stringify(response));
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const address = server.address();
if (!address || typeof address === "string")
throw new Error("Missing server address");
try {
const catalog = await connectCatalog(`http://127.0.0.1:${address.port}`, {
apiKey: "secret",
sqlHostOverride: "grpc+tls://sql.example.com:10026",
clientConfig: {
extraHeaders: {
"X-LanceDB-Database": "wrong-static",
"X-LanceDB-Database-Prefix": "wrong",
},
},
headerProvider: () => ({
"X-LanceDB-Database": "wrong-dynamic",
"X-LanceDB-Database-Prefix": "wrong",
authorization: "Bearer refreshed",
}),
});
await callback(catalog, requests);
expect(responses).toHaveLength(0);
} finally {
server.closeAllConnections();
await new Promise<void>((resolve) => server.close(() => resolve()));
}
}
describe("remote catalog", () => {
it("uses root namespace routes and preserves independent database scope", async () => {
await withCatalog(
[
[204, null],
[200, { tables: [] }],
[200, {}],
[200, { tables: [] }],
[200, { tables: [] }],
// biome-ignore lint/style/useNamingConvention: server wire format
[200, { namespaces: ["team/search"], page_token: "next" }],
[204, null],
],
async (catalog, requests) => {
const first = await catalog.createDatabase("team/search", {
existOk: true,
});
expect(await first.tableNames()).toEqual([]);
const second = await catalog.connectDatabase("other");
expect(await second.tableNames()).toEqual([]);
expect(await first.tableNames()).toEqual([]);
expect(
await catalog.listDatabases({ limit: 1, pageToken: "a/b" }),
).toEqual({ databases: ["team/search"], pageToken: "next" });
await catalog.dropDatabase("team/search", { ignoreMissing: true });
expect(requests[0].url).toBe("/v1/namespace/team%2Fsearch/create");
expect(requests[0].body).toEqual({ mode: "ExistOk" });
expect(requests[5].url).toBe(
"/v1/namespace/$/list?limit=1&page_token=a%2Fb",
);
expect(requests[6].body).toEqual({
mode: "Skip",
behavior: "Restrict",
});
for (const [i, request] of requests.entries()) {
expect(request.headers["x-lancedb-database"]).toBe(
i === 1 || i === 4 ? "team/search" : i === 3 ? "other" : undefined,
);
expect(request.headers["x-lancedb-database-prefix"]).toBeUndefined();
expect(request.headers.authorization).toBe("Bearer refreshed");
}
},
);
});
it("propagates lifecycle errors and sends restricted drops", async () => {
await withCatalog(
[
[404, {}],
[409, {}],
[400, {}],
[404, {}],
],
async (catalog, requests) => {
await expect(catalog.connectDatabase("missing")).rejects.toThrow(
"missing",
);
await expect(catalog.createDatabase("exists")).rejects.toThrow(
"exists",
);
await expect(catalog.dropDatabase("full")).rejects.toThrow();
await catalog.dropDatabase("missing", { ignoreMissing: true });
expect(requests[2].body).toEqual({
mode: "Fail",
behavior: "Restrict",
});
},
);
});
it("validates endpoints and pagination", async () => {
await expect(connectCatalog("/tmp/catalog")).rejects.toThrow();
const catalog = await connectCatalog("http://127.0.0.1:1");
for (const limit of [0, -1, 1.5, 2147483648]) {
await expect(catalog.listDatabases({ limit })).rejects.toThrow("limit");
}
await expect(catalog.connectDatabase("a$b")).rejects.toThrow(
"Invalid database name",
);
});
});