mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-21 12:35:42 +00:00
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
126 lines
4.0 KiB
Rust
126 lines
4.0 KiB
Rust
// SPDX-License-Identifier: Apache-2.0
|
|
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
|
|
|
|
use std::time::Duration;
|
|
|
|
use lancedb::catalog::{
|
|
CatalogConnection, CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
|
|
};
|
|
use pyo3::exceptions::PyValueError;
|
|
use pyo3::{Bound, PyAny, PyRef, PyResult, Python, pyclass, pyfunction, pymethods};
|
|
|
|
use crate::connection::{Connection, PyClientConfig};
|
|
use crate::error::PythonErrorExt;
|
|
use crate::runtime::future_into_py;
|
|
|
|
#[pyclass]
|
|
pub struct Catalog {
|
|
inner: CatalogConnection,
|
|
}
|
|
|
|
#[pymethods]
|
|
impl Catalog {
|
|
#[getter]
|
|
fn uri(&self) -> &str {
|
|
self.inner.uri()
|
|
}
|
|
|
|
#[pyo3(signature = (name, *, exist_ok=false))]
|
|
fn create_database<'py>(
|
|
self_: PyRef<'py, Self>,
|
|
name: String,
|
|
exist_ok: bool,
|
|
) -> PyResult<Bound<'py, PyAny>> {
|
|
let inner = self_.inner.clone();
|
|
future_into_py(self_.py(), async move {
|
|
inner
|
|
.create_database(CreateDatabaseRequest::new(name).exist_ok(exist_ok))
|
|
.await
|
|
.map(Connection::new)
|
|
.infer_error()
|
|
})
|
|
}
|
|
|
|
fn connect_database<'py>(self_: PyRef<'py, Self>, name: String) -> PyResult<Bound<'py, PyAny>> {
|
|
let inner = self_.inner.clone();
|
|
future_into_py(self_.py(), async move {
|
|
inner
|
|
.connect_database(name)
|
|
.await
|
|
.map(Connection::new)
|
|
.infer_error()
|
|
})
|
|
}
|
|
|
|
#[pyo3(signature = (name, *, ignore_missing=false))]
|
|
fn drop_database<'py>(
|
|
self_: PyRef<'py, Self>,
|
|
name: String,
|
|
ignore_missing: bool,
|
|
) -> PyResult<Bound<'py, PyAny>> {
|
|
let inner = self_.inner.clone();
|
|
future_into_py(self_.py(), async move {
|
|
inner
|
|
.drop_database(DropDatabaseRequest::new(name).ignore_missing(ignore_missing))
|
|
.await
|
|
.infer_error()
|
|
})
|
|
}
|
|
|
|
#[pyo3(signature = (*, limit=None, page_token=None))]
|
|
fn list_databases<'py>(
|
|
self_: PyRef<'py, Self>,
|
|
limit: Option<u32>,
|
|
page_token: Option<String>,
|
|
) -> PyResult<Bound<'py, PyAny>> {
|
|
let inner = self_.inner.clone();
|
|
future_into_py(self_.py(), async move {
|
|
let mut request = ListDatabasesRequest::default();
|
|
request.limit = limit;
|
|
request.page_token = page_token;
|
|
let response = inner.list_databases(request).await.infer_error()?;
|
|
Ok((response.databases, response.page_token))
|
|
})
|
|
}
|
|
}
|
|
|
|
#[pyfunction]
|
|
#[pyo3(signature = (endpoint, *, api_key=None, client_config=None, sql_host_override=None, read_consistency_interval=None, oauth_config=None))]
|
|
pub fn connect_catalog(
|
|
py: Python<'_>,
|
|
endpoint: String,
|
|
api_key: Option<String>,
|
|
client_config: Option<PyClientConfig>,
|
|
sql_host_override: Option<String>,
|
|
read_consistency_interval: Option<f64>,
|
|
oauth_config: Option<crate::oauth::PyOAuthConfig>,
|
|
) -> PyResult<Bound<'_, PyAny>> {
|
|
let interval = read_consistency_interval
|
|
.map(Duration::try_from_secs_f64)
|
|
.transpose()
|
|
.map_err(|err| {
|
|
PyValueError::new_err(format!("Invalid read consistency interval: {err}"))
|
|
})?;
|
|
future_into_py(py, async move {
|
|
let mut builder = lancedb::connect_catalog(endpoint);
|
|
if let Some(api_key) = api_key {
|
|
builder = builder.api_key(api_key);
|
|
}
|
|
if let Some(config) = client_config {
|
|
builder = builder.client_config(config.into());
|
|
}
|
|
if let Some(endpoint) = sql_host_override {
|
|
builder = builder.sql_host_override(endpoint);
|
|
}
|
|
if let Some(interval) = interval {
|
|
builder = builder.read_consistency_interval(interval);
|
|
}
|
|
if let Some(config) = oauth_config {
|
|
builder = builder.oauth_config(config.try_into().infer_error()?);
|
|
}
|
|
Ok(Catalog {
|
|
inner: builder.execute().await.infer_error()?,
|
|
})
|
|
})
|
|
}
|