mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
One page was carrying two layers. The catalog answers what exists and what it is called -- names, hierarchy, resolution, and the grants that follow the hierarchy. The lakehouse is everything that acts on those names: storing rows, searching them, deriving new ones, and keeping it all fast. Separating them makes the dependency visible. The catalog is a specification another engine can implement, which is why a Lance table is reachable without LanceDB; the lakehouse is where the cost and the tuning live. Governance belongs to the catalog because the hierarchy is what a grant attaches to, and maintenance belongs to the lakehouse because it is the files being rewritten. Also fixes a corruption from an earlier bulk link rewrite: "central metadata/catalog services" in prose had become a path.
364 lines
15 KiB
Plaintext
364 lines
15 KiB
Plaintext
---
|
|
title: "Namespaces"
|
|
sidebarTitle: "Namespace"
|
|
description: "Grouping tables into a hierarchy, and addressing them through it."
|
|
icon: folder-tree
|
|
keywords: ["namespace", "catalog", "hierarchy", "directory namespace", "rest namespace"]
|
|
---
|
|
|
|
import {
|
|
PyNamespaceTableOps,
|
|
PyNamespaceAdminOps,
|
|
TsNamespaceTableOps,
|
|
TsNamespaceAdminOps,
|
|
RsNamespaceAdminOps,
|
|
RsNamespaceTableOps,
|
|
} from '/snippets/connection.mdx';
|
|
|
|
Despite its name, LanceDB is not a "database" in the traditional sense -- it is a **Multimodal Lakehouse** that builds on the table abstraction,
|
|
similar to many other lakehouse projects. LanceDB exposes a catalog-level abstraction over the Lance table format, via a *namespace spec*.
|
|
If you're coming from traditional databases or lakehouses, you can think of a namespace as the catalog path that says where a table name lives.
|
|
|
|
Lance provides the **file** and **table** formats to store and manage your data and indexes.
|
|
LanceDB operates at the catalog layer (used to organize, discover, and operate on many Lance tables)
|
|
and provides a compute engine on top of the Lance format.
|
|
|
|
This is why many SDK methods in LanceDB, like `create_table`, `open_table`, `drop_table`, and
|
|
`rename_table`, accept namespace input. The SDK methods expose that input in the idiom of each
|
|
language: Python uses `namespace_path`, Rust uses builder methods like `.namespace(...)`, and
|
|
TypeScript uses `namespacePath` arguments.
|
|
|
|
<Note>
|
|
In SQL a namespaced table is addressed as a [dotted path](/sql/select#namespaces).
|
|
</Note>
|
|
|
|
## Namespace hierarchy {#namespace-hierarchy}
|
|
|
|
Namespaces are generalizations of catalog specs that give platform developers a clean way to present Lance tables in the structures users expect. The diagram below shows how the hierarchy can go beyond a single level.
|
|
A namespace can contain a collection of tables, and it can also contain namespaces recursively.
|
|
|
|

|
|
|
|
Before diving into examples, it helps to keep two terms in mind: the **namespace client** is the abstraction that presents a consistent namespace API, while the **namespace implementation** is the concrete backend that resolves namespaces and table locations (for example, a local directory or an external catalog).
|
|
If you want to go deeper, see the Lance format [namespace documentation](https://lance.org/format/namespace/).
|
|
|
|
## Namespace paths and names {#namespace-paths-and-names}
|
|
|
|
A namespace path is a list of components. For example, `["prod", "search"]` means the `search`
|
|
namespace inside the `prod` namespace. The empty path, `[]`, means the root namespace.
|
|
|
|
Each component is a name, not a filesystem path segment. Namespace names can't be empty, and each
|
|
component can contain only letters, numbers, underscores, hyphens, and periods. That keeps the same
|
|
identifier usable across local directory namespaces and REST namespace identifiers.
|
|
|
|
## Directory namespaces {#directory-namespaces}
|
|
|
|
The simplest namespace model in LanceDB is a single root namespace, often represented by one
|
|
directory:
|
|
|
|
```bash
|
|
./local_lancedb (root)
|
|
└── prod
|
|
└── search
|
|
└── user (table)
|
|
└── data (table)
|
|
```
|
|
|
|
As a user of LanceDB OSS, you might never notice namespaces at first, because LanceDB exposes the single-level hierarchy shown above, with the data stored in the `data/` directory, where the root namespace is implicit. Connecting to this namespace is as simple as connecting to the catalog root:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
import lancedb
|
|
|
|
# Connect to the directory namespace root
|
|
db = lancedb.connect("./local_lancedb")
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
import * as lancedb from "@lancedb/lancedb";
|
|
|
|
// Connect to the directory namespace root
|
|
const db = await lancedb.connect("./local_lancedb");
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
use lancedb::connect;
|
|
|
|
// Connect to the directory namespace root
|
|
let db = connect("./local_lancedb").execute().await?;
|
|
```
|
|
</CodeGroup>
|
|
|
|
This creates the default namespace directory (`data/`) under the specified root path.
|
|
|
|
You can also explicitly connect to a namespace using `lancedb.connect_namespace(...)` with the directory namespace implementation:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
import lancedb
|
|
|
|
# Local namespace-backed catalog root (DirectoryNamespace)
|
|
# See https://lance.org/format/namespace/dir/catalog-spec/
|
|
db = lancedb.connect_namespace("dir", {"root": "./local_lancedb"})
|
|
|
|
table_name = "user"
|
|
data = [{"id": 1, "vector": [0.1, 0.2], "name": "alice"}]
|
|
|
|
table = db.create_table(table_name, data=data, mode="create")
|
|
print(f"Created table: {table.name}")
|
|
# Created table: user
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
import * as lancedb from "@lancedb/lancedb";
|
|
|
|
// Local namespace-backed catalog root (DirectoryNamespace)
|
|
// See https://lance.org/format/namespace/dir/catalog-spec/
|
|
const db = await lancedb.connectNamespace("dir", { root: "./local_lancedb" });
|
|
|
|
const table = await db.createTable(
|
|
"user",
|
|
[{ id: 1, vector: [0.1, 0.2], name: "alice" }],
|
|
{ mode: "create" },
|
|
);
|
|
console.log(`Created table: ${table.name}`);
|
|
// Created table: user
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
use std::collections::HashMap;
|
|
use std::sync::Arc;
|
|
use arrow_schema::{DataType, Field, Schema};
|
|
|
|
// Local namespace-backed catalog root (DirectoryNamespace)
|
|
// See https://lance.org/format/namespace/dir/catalog-spec/
|
|
let mut properties = HashMap::new();
|
|
properties.insert("root".to_string(), "./local_lancedb".to_string());
|
|
let db = lancedb::connect_namespace("dir", properties)
|
|
.execute()
|
|
.await?;
|
|
|
|
let schema = Arc::new(Schema::new(vec![
|
|
Field::new("id", DataType::Int64, false),
|
|
]));
|
|
let table = db
|
|
.create_empty_table("user", schema)
|
|
.execute()
|
|
.await?;
|
|
println!("Created table: {}", table.name());
|
|
// Created table: user
|
|
```
|
|
</CodeGroup>
|
|
|
|
<Info>
|
|
- For simple use cases in LanceDB OSS, you don't need to go too deep into namespaces.
|
|
- To integrate LanceDB with external catalogs and to use it as a true **multimodal lakehouse**, it's useful to understand the different namespace implementations and how to use them in your organization's setup.
|
|
</Info>
|
|
|
|
## Remote or external catalog namespaces {#remote-or-external-catalog-namespaces}
|
|
|
|
The example above showed local directory-based namespaces. LanceDB also supports namespaces backed by remote object stores and external catalogs, via the REST namespace implementation.
|
|
|
|
For remote object stores with central metadata or catalog services (either commercial or open source),
|
|
use the REST namespace implementation. It is backed by REST routes
|
|
(for example `POST /v1/namespace/{id}/create` and `GET /v1/namespace/{id}/list`) and server-provided table locations.
|
|
|
|
For authentication, any property prefixed with `headers` is forwarded as an HTTP header
|
|
(for example `headers.Authorization` becomes `Authorization`, and `headers.X-API-Key` becomes `X-API-Key`).
|
|
LanceDB Enterprise REST requests use the `x-api-key` header for API-key authentication. Deployments
|
|
that route multiple databases through the same endpoint can also use headers such as
|
|
`x-lancedb-database` or `x-lancedb-database-prefix` for database context.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
import os
|
|
import lancedb
|
|
|
|
# Remote namespace-backed catalog root (RestNamespace)
|
|
# See https://lance.org/format/namespace/rest/catalog-spec/
|
|
db = lancedb.connect_namespace(
|
|
"rest",
|
|
{
|
|
"uri": "https://<your_catalog>.internal.<your_org>.com",
|
|
"headers.x-api-key": os.environ["API_KEY"],
|
|
# or:
|
|
# "headers.Authorization": f"Bearer {os.environ['REST_AUTH_TOKEN']}",
|
|
},
|
|
)
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
import * as lancedb from "@lancedb/lancedb";
|
|
|
|
// Remote namespace-backed catalog root (RestNamespace)
|
|
// See https://lance.org/format/namespace/rest/catalog-spec/
|
|
const db = await lancedb.connectNamespace("rest", {
|
|
uri: "https://<your_catalog>.internal.<your_org>.com",
|
|
headers: {
|
|
"x-api-key": process.env.API_KEY ?? "",
|
|
// or:
|
|
// Authorization: `Bearer ${process.env.REST_AUTH_TOKEN}`,
|
|
},
|
|
});
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
use std::collections::HashMap;
|
|
|
|
// Remote namespace-backed catalog root (RestNamespace)
|
|
// See https://lance.org/format/namespace/rest/catalog-spec/
|
|
let mut properties = HashMap::new();
|
|
properties.insert(
|
|
"uri".to_string(),
|
|
"https://<your_catalog>.internal.<your_org>.com".to_string(),
|
|
);
|
|
properties.insert(
|
|
"headers.x-api-key".to_string(),
|
|
std::env::var("API_KEY")?,
|
|
);
|
|
// or:
|
|
// properties.insert(
|
|
// "headers.Authorization".to_string(),
|
|
// format!("Bearer {}", std::env::var("REST_AUTH_TOKEN")?),
|
|
// );
|
|
let db = lancedb::connect_namespace("rest", properties)
|
|
.execute()
|
|
.await?;
|
|
```
|
|
</CodeGroup>
|
|
[LanceDB Enterprise](/enterprise) operates a REST namespace server on top of the Lance format, so any REST client that can speak the REST namespace API
|
|
contract can be used to interact with it. For authentication examples in LanceDB Enterprise, visit
|
|
the [Namespaces in SDKs](#namespaces-in-lancedb-enterprise) page.
|
|
|
|
## Best practices {#best-practices}
|
|
|
|
Below, we list some best practices for working with namespaces:
|
|
- For simple use cases and single, stand-alone applications, the directory-based root namespace is sufficient and requires no special configuration.
|
|
- For remote storage locations, introduce explicit namespaces when multiple teams, environments, or domains share the same catalog.
|
|
- Treat namespace paths as stable identifiers (for example `"prod/search"`, `"staging/recs"`).
|
|
- For maintainability reasons, avoid hard-coding object-store table paths in application code -- instead, prefer catalog identifiers + namespaces.
|
|
|
|
|
|
## Table operations with namespace paths {#table-operations-with-namespace-paths}
|
|
|
|
Let's imagine a scenario where your table management needs have evolved, and you now have the following multi-level structure to organize your tables outside the root namespace.
|
|
```
|
|
./local_lancedb (root)
|
|
└── prod
|
|
└── search
|
|
└── user (table)
|
|
└── data (table)
|
|
└── recommendations
|
|
└── user (table)
|
|
└── data (data)
|
|
```
|
|
|
|
Below, we show how you would express table operations within that namespace. Each item in the namespace
|
|
list (`["prod", "search"]`) represents a level in the namespace hierarchy, and the table name is
|
|
specified when you create, open, list, or drop it.
|
|
|
|
The SDK methods expose the namespace path in the idiom of each language:
|
|
|
|
- Python: pass `namespace_path=["prod", "search"]` to table operations.
|
|
- Rust: call builder methods such as `.namespace(vec!["prod".to_string(), "search".to_string()])`.
|
|
- TypeScript: pass a `namespacePath` array, for example `await db.openTable("user", ["prod", "search"])`.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyNamespaceTableOps}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsNamespaceTableOps}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsNamespaceTableOps}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<Note>
|
|
Using namespaces is **optional** in LanceDB, and most basic use cases do not require to work with them.
|
|
An empty namespace (`[]`), which is the default, means "root namespace", and the data will be stored in
|
|
the `data/` directory under the specified root path.
|
|
</Note>
|
|
|
|
## Namespace management APIs {#namespace-management-apis}
|
|
|
|
You can open/create/drop tables inside a namespace path (like `["prod", "search"]`).
|
|
All three SDKs expose namespace lifecycle operations directly.
|
|
In Python, use `lancedb.connect_namespace(...)` when calling namespace lifecycle methods such as
|
|
`create_namespace`, `list_namespaces`, `describe_namespace`, and `drop_namespace`.
|
|
In TypeScript, use `lancedb.connectNamespace(...)` and call `createNamespace`, `listNamespaces`,
|
|
`describeNamespace`, and `dropNamespace` on the returned `Connection`.
|
|
In Rust, use `lancedb::connect_namespace(...)` and call `create_namespace`, `list_namespaces`,
|
|
and `drop_namespace`.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyNamespaceAdminOps}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsNamespaceAdminOps}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsNamespaceAdminOps}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Namespace creation and deletion have modes that control what happens when the target already exists,
|
|
doesn't exist, or contains data:
|
|
|
|
- Create mode: `create` fails if the namespace already exists, `exist_ok` keeps the existing namespace, and `overwrite` replaces it.
|
|
- Drop mode: `fail` reports an error when the namespace doesn't exist, and `skip` treats a missing namespace as a successful no-op.
|
|
- Drop behavior: `restrict` keeps non-empty namespaces from being dropped, and `cascade` drops child namespaces and tables first.
|
|
|
|
Namespace path components can't be empty. Each component can contain only letters, numbers,
|
|
underscores, hyphens, and periods.
|
|
|
|
Listing APIs return the immediate children of the requested namespace path. Use `limit` with the
|
|
returned `page_token` to page through large catalogs; pass an empty namespace path (`[]`) when you
|
|
want to list from the root namespace.
|
|
|
|
## Namespaces in LanceDB Enterprise {#namespaces-in-lancedb-enterprise}
|
|
|
|
In LanceDB Enterprise deployments, configure namespace-backed federated databases in a TOML file under your deployment's `config` directory.
|
|
LanceDB Enterprise supports both directory-based (`ns_impl = "dir"`) and REST-based (`ns_impl = "rest"`) namespace implementations.
|
|
The example below shows how to configure a directory-based namespace implementation in LanceDB Enterprise.
|
|
```toml
|
|
# Federated database configuration for DirectoryNamespace
|
|
# This example uses minio storage
|
|
[federated_dbs.federated_dir_test]
|
|
ns_impl = "dir"
|
|
root = "s3://<your-deployment>/<your-dir>"
|
|
"storage.region" = "us-east-1"
|
|
"storage.endpoint" = "http://localhost:9000"
|
|
"storage.access_key_id" = "minioadmin"
|
|
"storage.secret_access_key" = "minioadmin"
|
|
"storage.allow_http" = "true"
|
|
# Far future expiration (year 2100)
|
|
"storage.expires_at_millis" = "4102444800000"
|
|
```
|
|
|
|
The example above uses MinIO, but the same approach applies to other cloud object storage platforms based on your deployment.
|
|
|
|
For REST-based namespace servers, you can specify the namespace implementation as `"rest"` with forwarding prefixed headers
|
|
for authentication and context propagation.
|
|
|
|
```toml
|
|
[federated_dbs.federated_rest_test]
|
|
ns_impl = "rest"
|
|
uri = "http://<your_org>.internal.catalog.com"
|
|
forward_header_prefixes = ["X-forward"]
|
|
```
|
|
|
|
With `forward_header_prefixes = ["X-forward"]`, any incoming header starting with `X-forward` is forwarded to
|
|
`http://<your_org>.internal.catalog.com`. This is useful for auth propagation, for example sending
|
|
`X-forward-authorization: Bearer xxxx`.
|
|
|
|
For the LanceDB REST API itself, requests use `x-api-key` for API-key authentication. If your endpoint
|
|
serves more than one database, LanceDB can also use headers such as `x-lancedb-database` or
|
|
`x-lancedb-database-prefix` to route the request to the right database context.
|