mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-22 04:55:39 +00:00
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
143 lines
4.9 KiB
TypeScript
143 lines
4.9 KiB
TypeScript
// SPDX-License-Identifier: Apache-2.0
|
|
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
|
|
|
|
import * as http from "http";
|
|
import { Catalog, connectCatalog } from "../lancedb";
|
|
|
|
type RecordedRequest = {
|
|
url: string;
|
|
headers: http.IncomingHttpHeaders;
|
|
body: Record<string, unknown>;
|
|
};
|
|
|
|
async function withCatalog(
|
|
responses: [number, unknown][],
|
|
callback: (catalog: Catalog, requests: RecordedRequest[]) => Promise<void>,
|
|
) {
|
|
const requests: RecordedRequest[] = [];
|
|
const server = http.createServer(async (req, res) => {
|
|
const chunks: Buffer[] = [];
|
|
for await (const chunk of req) chunks.push(Buffer.from(chunk));
|
|
const body = Buffer.concat(chunks).toString();
|
|
requests.push({
|
|
url: req.url ?? "",
|
|
headers: req.headers,
|
|
body: body ? JSON.parse(body) : {},
|
|
});
|
|
const [status, response] = responses.shift() ?? [
|
|
500,
|
|
{ error: "Unexpected request" },
|
|
];
|
|
res.writeHead(status, { "content-type": "application/json" });
|
|
res.end(status === 204 ? undefined : JSON.stringify(response));
|
|
});
|
|
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
|
|
const address = server.address();
|
|
if (!address || typeof address === "string")
|
|
throw new Error("Missing server address");
|
|
try {
|
|
const catalog = await connectCatalog(`http://127.0.0.1:${address.port}`, {
|
|
apiKey: "secret",
|
|
sqlHostOverride: "grpc+tls://sql.example.com:10026",
|
|
clientConfig: {
|
|
extraHeaders: {
|
|
"X-LanceDB-Database": "wrong-static",
|
|
"X-LanceDB-Database-Prefix": "wrong",
|
|
},
|
|
},
|
|
headerProvider: () => ({
|
|
"X-LanceDB-Database": "wrong-dynamic",
|
|
"X-LanceDB-Database-Prefix": "wrong",
|
|
authorization: "Bearer refreshed",
|
|
}),
|
|
});
|
|
await callback(catalog, requests);
|
|
expect(responses).toHaveLength(0);
|
|
} finally {
|
|
server.closeAllConnections();
|
|
await new Promise<void>((resolve) => server.close(() => resolve()));
|
|
}
|
|
}
|
|
|
|
describe("remote catalog", () => {
|
|
it("uses root namespace routes and preserves independent database scope", async () => {
|
|
await withCatalog(
|
|
[
|
|
[204, null],
|
|
[200, { tables: [] }],
|
|
[200, {}],
|
|
[200, { tables: [] }],
|
|
[200, { tables: [] }],
|
|
// biome-ignore lint/style/useNamingConvention: server wire format
|
|
[200, { namespaces: ["team/search"], page_token: "next" }],
|
|
[204, null],
|
|
],
|
|
async (catalog, requests) => {
|
|
const first = await catalog.createDatabase("team/search", {
|
|
existOk: true,
|
|
});
|
|
expect(await first.tableNames()).toEqual([]);
|
|
const second = await catalog.connectDatabase("other");
|
|
expect(await second.tableNames()).toEqual([]);
|
|
expect(await first.tableNames()).toEqual([]);
|
|
expect(
|
|
await catalog.listDatabases({ limit: 1, pageToken: "a/b" }),
|
|
).toEqual({ databases: ["team/search"], pageToken: "next" });
|
|
await catalog.dropDatabase("team/search", { ignoreMissing: true });
|
|
expect(requests[0].url).toBe("/v1/namespace/team%2Fsearch/create");
|
|
expect(requests[0].body).toEqual({ mode: "ExistOk" });
|
|
expect(requests[5].url).toBe(
|
|
"/v1/namespace/%24/list?limit=1&page_token=a%2Fb",
|
|
);
|
|
expect(requests[6].body).toEqual({
|
|
mode: "Skip",
|
|
behavior: "Restrict",
|
|
});
|
|
for (const [i, request] of requests.entries()) {
|
|
expect(request.headers["x-lancedb-database"]).toBe(
|
|
i === 1 || i === 4 ? "team/search" : i === 3 ? "other" : undefined,
|
|
);
|
|
expect(request.headers["x-lancedb-database-prefix"]).toBeUndefined();
|
|
expect(request.headers.authorization).toBe("Bearer refreshed");
|
|
}
|
|
},
|
|
);
|
|
});
|
|
|
|
it("propagates lifecycle errors and sends restricted drops", async () => {
|
|
await withCatalog(
|
|
[
|
|
[404, {}],
|
|
[409, {}],
|
|
[400, {}],
|
|
[404, {}],
|
|
],
|
|
async (catalog, requests) => {
|
|
await expect(catalog.connectDatabase("missing")).rejects.toThrow(
|
|
"missing",
|
|
);
|
|
await expect(catalog.createDatabase("exists")).rejects.toThrow(
|
|
"exists",
|
|
);
|
|
await expect(catalog.dropDatabase("full")).rejects.toThrow();
|
|
await catalog.dropDatabase("missing", { ignoreMissing: true });
|
|
expect(requests[2].body).toEqual({
|
|
mode: "Fail",
|
|
behavior: "Restrict",
|
|
});
|
|
},
|
|
);
|
|
});
|
|
|
|
it("validates endpoints and pagination", async () => {
|
|
await expect(connectCatalog("/tmp/catalog")).rejects.toThrow();
|
|
const catalog = await connectCatalog("http://127.0.0.1:1");
|
|
for (const limit of [0, -1, 1.5, 2147483648]) {
|
|
await expect(catalog.listDatabases({ limit })).rejects.toThrow("limit");
|
|
}
|
|
await expect(catalog.connectDatabase("a$b")).rejects.toThrow(
|
|
"Invalid database name",
|
|
);
|
|
});
|
|
});
|