mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-30 17:05:49 +00:00
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
104 lines
3.2 KiB
TypeScript
104 lines
3.2 KiB
TypeScript
// SPDX-License-Identifier: Apache-2.0
|
|
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
|
|
|
|
import { Connection, LocalConnection } from "./connection";
|
|
import { HeaderProvider } from "./header";
|
|
import {
|
|
JsHeaderProvider,
|
|
ListDatabasesResponse,
|
|
Catalog as NativeCatalog,
|
|
CatalogOptions as NativeCatalogOptions,
|
|
} from "./native.js";
|
|
import { OAuthConfig } from "./oauth";
|
|
|
|
/** Options shared by a catalog and the database connections it returns. */
|
|
export interface CatalogOptions
|
|
extends Omit<NativeCatalogOptions, "oauthConfig"> {
|
|
oauthConfig?: OAuthConfig;
|
|
/** Called for each request to supply authentication headers. */
|
|
headerProvider?:
|
|
| HeaderProvider
|
|
| (() => Record<string, string> | Promise<Record<string, string>>);
|
|
}
|
|
|
|
export type { ListDatabasesResponse } from "./native.js";
|
|
|
|
/** A remote catalog manages databases through the server's root namespace. */
|
|
export class Catalog {
|
|
/** @hidden */
|
|
constructor(private readonly inner: NativeCatalog) {}
|
|
|
|
/** The root namespace endpoint. */
|
|
get uri(): string {
|
|
return this.inner.uri;
|
|
}
|
|
|
|
/** Create a database, or open an existing database when existOk is true. */
|
|
async createDatabase(
|
|
name: string,
|
|
options: { existOk?: boolean } = {},
|
|
): Promise<Connection> {
|
|
return new LocalConnection(
|
|
await this.inner.createDatabase(name, options.existOk),
|
|
);
|
|
}
|
|
|
|
/** Connect to an existing database by its logical name. */
|
|
async connectDatabase(name: string): Promise<Connection> {
|
|
return new LocalConnection(await this.inner.connectDatabase(name));
|
|
}
|
|
|
|
/** Drop an empty database. The server rejects nonempty databases. */
|
|
async dropDatabase(
|
|
name: string,
|
|
options: { ignoreMissing?: boolean } = {},
|
|
): Promise<void> {
|
|
await this.inner.dropDatabase(name, options.ignoreMissing);
|
|
}
|
|
|
|
/** List one page of databases; pass pageToken from a response for the next page. */
|
|
async listDatabases(
|
|
options: { limit?: number; pageToken?: string } = {},
|
|
): Promise<ListDatabasesResponse> {
|
|
if (
|
|
options.limit !== undefined &&
|
|
(!Number.isInteger(options.limit) ||
|
|
options.limit <= 0 ||
|
|
options.limit > 2147483647)
|
|
) {
|
|
throw new Error(
|
|
"Database list limit must be an integer between 1 and 2147483647",
|
|
);
|
|
}
|
|
return this.inner.listDatabases(options.limit, options.pageToken);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Connect to an HTTP(S) catalog endpoint. Catalog requests omit database-selection
|
|
* headers; opened database connections inherit authentication and client options.
|
|
*
|
|
* @example
|
|
* ```ts
|
|
* const catalog = await connectCatalog("https://my-server.example", { apiKey: "secret" });
|
|
* const db = await catalog.createDatabase("analytics", { existOk: true });
|
|
* const page = await catalog.listDatabases({ limit: 20 });
|
|
* ```
|
|
*/
|
|
export async function connectCatalog(
|
|
endpoint: string,
|
|
options: CatalogOptions = {},
|
|
): Promise<Catalog> {
|
|
const { headerProvider, ...nativeOptions } = options;
|
|
const provider = headerProvider
|
|
? new JsHeaderProvider(async () =>
|
|
typeof headerProvider === "function"
|
|
? headerProvider()
|
|
: headerProvider.getHeaders(),
|
|
)
|
|
: undefined;
|
|
return new Catalog(
|
|
await NativeCatalog.new(endpoint, nativeOptions, provider),
|
|
);
|
|
}
|