feat: add remote catalogs and Python and TypeScript bindings (#4195)

Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.

## Rust API

`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.

```rust
use lancedb::catalog::{
    CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};

let catalog = lancedb::connect_catalog("https://my-server.example")
    .api_key("my-api-key")
    .execute()
    .await?;

let db = catalog.create_database(
    CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
    ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
    DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```

Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.

## Python API

```python
import lancedb

catalog = lancedb.connect_catalog(
    "https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
    next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```

`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.

## TypeScript API

```typescript
import { connectCatalog } from "@lancedb/lancedb";

const catalog = await connectCatalog("https://my-server.example", {
  apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
  const nextPage = await catalog.listDatabases({
    limit: 20, pageToken: page.pageToken,
  });
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```

Create/connect return the existing `Connection` API. All four methods
are asynchronous.

## REST mapping

All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.

| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |

For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.

Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.

For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.

Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.

Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.

## Scope

This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.

The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>

python/python/tests/test_catalog.py

@@ -0,0 +1,140 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+import json
+from threading import Thread
+
+import pytest
+
+import lancedb
+from lancedb.db import AsyncConnection, DBConnection
+from lancedb.remote.errors import HttpError
+
+
+@pytest.fixture
+def catalog_server():
+    requests = []
+    responses = []
+
+    class Handler(BaseHTTPRequestHandler):
+        def handle_request(self):
+            body = self.rfile.read(int(self.headers.get("Content-Length", "0")))
+            requests.append(
+                (
+                    self.path,
+                    dict(self.headers.items()),
+                    json.loads(body) if body else None,
+                )
+            )
+            status, response = responses.pop(0)
+            self.send_response(status)
+            self.send_header("Content-Type", "application/json")
+            self.end_headers()
+            if status != 204:
+                self.wfile.write(json.dumps(response).encode())
+
+        do_GET = handle_request
+        do_POST = handle_request
+
+    with ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
+        thread = Thread(target=server.serve_forever, daemon=True)
+        thread.start()
+        try:
+            yield f"http://127.0.0.1:{server.server_port}", requests, responses
+        finally:
+            server.shutdown()
+            thread.join()
+
+
+def test_catalog_sync_scope_and_serialization(catalog_server):
+    endpoint, requests, responses = catalog_server
+    responses.extend(
+        [
+            (204, None),
+            (200, {}),
+            (200, {"tables": []}),
+            (200, {"tables": []}),
+            (200, {"namespaces": ["team/search"], "page_token": "next"}),
+            (204, None),
+        ]
+    )
+    catalog = lancedb.connect_catalog(
+        endpoint,
+        api_key="secret",
+        sql_host_override="invalid://localhost",
+        client_config={
+            "extra_headers": {
+                "X-LanceDB-Database": "wrong",
+                "X-LanceDB-Database-Prefix": "wrong",
+            }
+        },
+    )
+    assert isinstance(catalog, lancedb.Catalog)
+    assert catalog.uri == endpoint
+    db = catalog.create_database("team/search", exist_ok=True)
+    assert isinstance(db, DBConnection)
+    with pytest.raises(ValueError, match="sql_host_override must use"):
+        db.execute_query_async("SELECT 1")
+    db = catalog.connect_database("team/search")
+    assert isinstance(db, DBConnection)
+    assert db.table_names() == []
+    restored = lancedb.deserialize_conn(db.serialize())
+    assert restored.sql_host_override == "invalid://localhost"
+    for connection in (db, restored):
+        with pytest.raises(ValueError, match="sql_host_override must use"):
+            connection.execute_query_async("SELECT 1")
+    assert restored.table_names() == []
+    page = catalog.list_databases(limit=1, page_token="a/b")
+    assert page == lancedb.ListDatabasesResponse(["team/search"], "next")
+    catalog.drop_database("team/search", ignore_missing=True)
+    assert requests[0][0] == "/v1/namespace/team%2Fsearch/create"
+    assert requests[0][2] == {"mode": "ExistOk"}
+    assert requests[1][0] == "/v1/namespace/team%2Fsearch/describe"
+    assert requests[4][0] == "/v1/namespace/%24/list?limit=1&page_token=a%2Fb"
+    assert requests[5][2] == {"mode": "Skip", "behavior": "Restrict"}
+    for i, (_, headers, _) in enumerate(requests):
+        headers = {key.lower(): value for key, value in headers.items()}
+        assert headers.get("x-lancedb-database") == (
+            "team/search" if i in (2, 3) else None
+        )
+        assert "x-lancedb-database-prefix" not in headers
+        assert headers["x-api-key"] == "secret"
+
+
+@pytest.mark.asyncio
+async def test_catalog_async_and_errors(catalog_server):
+    endpoint, requests, responses = catalog_server
+    responses.extend(
+        [
+            (200, {}),
+            (200, {"tables": []}),
+            (404, {"error": "missing"}),
+            (409, {"error": "exists"}),
+            (400, {"error": "not empty"}),
+            (404, {"error": "missing"}),
+        ]
+    )
+    catalog = await lancedb.connect_catalog_async(
+        endpoint, sql_host_override="invalid://localhost"
+    )
+    assert isinstance(catalog, lancedb.AsyncCatalog)
+    db = await catalog.connect_database("analytics")
+    assert isinstance(db, AsyncConnection)
+    with pytest.raises(ValueError, match="sql_host_override must use"):
+        await db.execute_query_async("SELECT 1")
+    assert await db.table_names() == []
+    with pytest.raises(ValueError, match="missing"):
+        await catalog.connect_database("missing")
+    with pytest.raises(ValueError, match="exists"):
+        await catalog.create_database("exists")
+    with pytest.raises(HttpError):
+        await catalog.drop_database("full")
+    await catalog.drop_database("missing", ignore_missing=True)
+    assert requests[4][2] == {"mode": "Fail", "behavior": "Restrict"}
+
+
+@pytest.mark.parametrize("endpoint", ["/tmp/catalog", "s3://bucket", "db://db"])
+def test_catalog_requires_remote_endpoint(endpoint):
+    with pytest.raises(ValueError, match="endpoint"):
+        lancedb.connect_catalog(endpoint)