rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-13 07:12:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: split Lance Catalog from Lance Lakehouse

Browse Source 
One page was carrying two layers. The catalog answers what exists and what it is
called -- names, hierarchy, resolution, and the grants that follow the
hierarchy. The lakehouse is everything that acts on those names: storing rows,
searching them, deriving new ones, and keeping it all fast.

Separating them makes the dependency visible. The catalog is a specification
another engine can implement, which is why a Lance table is reachable without
LanceDB; the lakehouse is where the cost and the tuning live. Governance belongs
to the catalog because the hierarchy is what a grant attaches to, and
maintenance belongs to the lakehouse because it is the files being rewritten.

Also fixes a corruption from an earlier bulk link rewrite: "central
metadata/catalog services" in prose had become a path.
This commit is contained in:
Jack Ye
2026-09-25 17:43:19 -07:00
parent f324b0713b
commit 5d1cb479e8
4 changed files with 167 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/basics/catalog.mdx
+63 -35
View File
@@ -1,18 +1,20 @@
---
title: "Lakehouse and catalog"
sidebarTitle: "Lakehouse & Catalog"
description: "What a multimodal lakehouse holds, and how the catalog organizes it."
title: "Lance Catalog"
sidebarTitle: "Lance Catalog"
description: "The naming layer: what objects exist, how they nest, and who may reach them."
icon: sitemap
keywords: ["lakehouse", "catalog", "database", "namespace", "table", "view", "function"]
keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view", "function"]
---
LanceDB is a **multimodal lakehouse**: data sits in open [Lance](/basics/lance)
tables on object storage, and a catalog gives those tables names, structure, and
the operations that act on them.
A catalog answers one question: **what exists, and what is it called.** It holds
names, the hierarchy they sit in, and the rules about who may reach them. It
does not store rows, run queries, or build indexes -- that is the
[lakehouse](/basics/lakehouse) on top of it.
The lakehouse part is why images, embeddings, and the columns you derive from
them live in one table rather than in a warehouse plus a vector store plus a
feature store. The catalog part is what this page is about.
Keeping the two apart is what lets a Lance table be reached by something that is
not LanceDB. The catalog is a specification -- the
[Lance Namespace](/basics/lance#standard) -- so any engine that implements it
resolves the same names to the same tables.
## The objects {#objects}
@@ -21,7 +23,7 @@ feature store. The catalog part is what this page is about.
| [Database](/management/database) | The root. A connection opens exactly one. |
| [Namespace](/management/namespace) | A named grouping inside a database. Can nest. |
| [Table](/management/table) | Rows and columns, versioned. What most operations act on. |
| [Materialized view](/feature-engineering/materialized-views) | A table whose contents come from a query, computed and stored. |
| [Materialized view](/management/materialized-view) | A table whose contents come from a query, computed and stored. |
| [View](/management/view) | A saved query. Computed when read, not stored. |
| [Function](/management/function) | Registered code, applied to produce column values. |
@@ -34,34 +36,46 @@ database
└── function
```
A namespace can contain further namespaces, so the hierarchy goes as deep as you
need. A table is reached by its path through it.
A namespace can contain further namespaces, so the hierarchy goes as deep as
your organisation needs. A table is reached by its path through it:
`prod.events.clicks`.
This page is about what the objects *are*. For the operations on them --
connecting, creating, listing, dropping -- see [Management](/management).
This page is about what the objects *are*. For the operations on them, see
[Management](/management).
## A table is more than rows {#tables}
## Resolution {#resolution}
A Lance table carries things a warehouse table does not, and most of the rest of
these docs is about one of them:
A name is resolved by walking the path, and the walk is the point: reaching a
table means traversing the namespaces above it. That is what makes the hierarchy
more than decoration -- it is the unit of both organisation and access.
| | |
| --- | --- |
| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/add-computed-column) from other columns |
| **Versions** | every write commits one, so reads can [travel back](/management/table#versioning) |
| **Indexes** | vector, full-text, and scalar, built over the columns |
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
A flat set of tables needs none of this. Namespaces start earning their keep when
several teams, environments or domains share one catalog and you want
`prod.search.documents` and `staging.search.documents` to be unambiguous.
That is why "add a feature column" is a normal operation here rather than a
migration: it writes one new column and leaves the expensive data alone.
## Governance {#governance}
<Badge color="red">Enterprise</Badge>
Because the catalog knows the hierarchy, it is the layer that can enforce
access. A grant names **one privilege** on **one object** for **one subject**,
and it inherits down the path: reaching a table requires `USAGE` on the
namespace containing it and on the database containing that.
That is the practical reason to put a boundary in the hierarchy -- a namespace
per team gives you something to grant on. See
[Authorization](/governance/authorization).
Embedded LanceDB has no such layer. Access is whatever the object store enforces
on the bucket, which has no concept of a table.
## Views and materialized views are different objects {#views}
The names are close enough to mislead. Both are defined by a query over other
tables; the difference is when that query runs.
The names are close enough to mislead. Both are catalog objects defined by a
query over other tables; the difference is when that query runs.
- A **materialized view** stores its rows. A refresh computes them, and reads are
reads of stored data -- fast, and as current as the last refresh.
- A **materialized view** stores its rows. A refresh computes them, and reads
are reads of stored data -- fast, and as current as the last refresh.
- A **view** stores only the query. Every read runs it, so a view is never stale
and never faster than the query it wraps.
@@ -69,10 +83,24 @@ Reach for a materialized view when the query is expensive and the data changes
less often than it is read. Reach for a view when correctness matters more than
latency, or when the query is cheap.
## What is not a catalog object {#elsewhere}
## What the catalog does not hold {#elsewhere}
**Indexes** belong to a table rather than standing alone, and creating one is a
performance decision -- see [Maintenance](/indexing).
**Indexes** belong to a table rather than standing alone. Creating one is a
performance decision, not a structural one -- see [Indexing](/indexing).
**Jobs** are not stored data at all; they are handles to work the server is
doing. See [Jobs](/basics/jobs).
**Jobs** are not stored data at all; they are handles to work in progress. See
[Jobs](/basics/jobs).
**Rows** are in the [Lance](/basics/lance) files the catalog points at. The
catalog knows a table exists and where it lives; it does not know what is in it.
## Next {#next}
<CardGroup cols={2}>
<Card title="Lance Lakehouse" icon="layer-group" href="/basics/lakehouse">
What operates on these objects: search, compute, maintenance.
</Card>
<Card title="Management" icon="folder-tree" href="/management">
Creating, listing and dropping each one.
</Card>
</CardGroup>
docs/web/basics/lakehouse.mdx
+102
View File
@@ -0,0 +1,102 @@
---
title: "Lance Lakehouse"
sidebarTitle: "Lance Lakehouse"
description: "What operates on the catalog's objects: storage, search, computation, and maintenance."
icon: layer-group
keywords: ["lakehouse", "multimodal", "search", "compute", "maintenance", "feature engineering"]
---
The [catalog](/basics/catalog) says what exists. The **lakehouse** is everything
that acts on it: storing the rows, searching them, computing new ones, and
keeping the whole thing fast as it grows.
The distinction matters because the two layers have different lifetimes. A
catalog entry is a name and a pointer. A lakehouse operation reads and writes
the [Lance](/basics/lance) files that name points at -- and those operations are
where almost all of the cost, and all of the tuning, lives.
## Multimodal is the point {#multimodal}
A lakehouse table holds images, video, audio, text, embeddings, annotations and
the features you derive from them **in one table**, side by side.
That is the difference from assembling a warehouse plus a vector store plus a
feature store. There is one copy of the data, one set of versions, and no
pipeline keeping three systems agreeing about the same rows. A search can filter
on a label, rank by a vector, and return the blob, without leaving the table.
| | |
| --- | --- |
| **Columns** | vectors, blobs, and columns [computed](/feature-engineering/add-computed-column) from other columns |
| **Versions** | every write commits one, so reads can [travel back](/management/table#versioning) |
| **Indexes** | vector, full-text and scalar, built over those columns |
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
## The four capabilities {#capabilities}
Everything the lakehouse does falls into one of four groups, and the rest of
these docs is organized the same way.
### Ingest {#ingest}
Getting rows in, and changing the ones that are there: append, merge, update,
delete. Every write commits a version rather than mutating in place, which is
what makes the other three safe to run against a moving table. See
[Data Ingestion](/ingestion).
### Search {#search}
Vector, full-text and hybrid retrieval over the same rows, with filters that
push into the scan rather than running after it. The search is a lakehouse
operation, not a separate service holding a second copy of your embeddings. See
[Semantic Search](/search).
### Compute {#compute}
Deriving new columns and tables from what is already stored:
[computed columns](/feature-engineering/add-computed-column),
[functions](/management/function) that run Python where the data is, and
[materialized views](/management/materialized-view) that persist a query's
result.
The reason this belongs in the lakehouse rather than in a pipeline outside it is
the format. Adding a derived column writes only that column -- the images and
videos are never rewritten -- so deriving a feature is an ordinary operation
rather than a migration. See [Feature Engineering](/feature-engineering) and
[Data Curation](/curation).
### Maintenance {#maintenance}
The work that keeps the first three fast: building and repairing
[indexes](/indexing), and [compaction](/maintenance/compaction) to merge small
files, prune old versions and fold new rows into existing indexes.
<Warning>
None of it runs on its own. There is no background compactor -- `optimize()`
runs when you call it, and an index does not cover rows added after it was
built until you do. See [Guarantees](/basics/guarantees#not-guaranteed).
</Warning>
## Where the work happens {#where}
The same lakehouse runs in two shapes, and which one you use decides where these
capabilities execute rather than whether you have them:
- **Embedded** -- your process does all four itself, directly against object
storage. Nothing runs unless you call it.
- **Distributed** -- a cluster does them, with indexing and compaction on
workers separate from the ones serving queries, plus
[SQL](/sql) and server-side [jobs](/basics/jobs).
See [Architecture](/architecture) for the comparison in full.
## Next {#next}
<CardGroup cols={2}>
<Card title="Lance Catalog" icon="sitemap" href="/basics/catalog">
The objects these operations act on.
</Card>
<Card title="Lance format" icon="layer-group" href="/basics/lance">
The open format underneath, and why column evolution is cheap.
</Card>
</CardGroup>
docs/web/docs.json
+1
View File
@@ -167,6 +167,7 @@
"pages": [
"basics/lance",
"basics/catalog",
"basics/lakehouse",
"basics/query",
"basics/jobs",
"tables/consistency",
docs/web/management/namespace.mdx
+1 -1
View File
@@ -159,7 +159,7 @@ println!("Created table: {}", table.name());
The example above showed local directory-based namespaces. LanceDB also supports namespaces backed by remote object stores and external catalogs, via the REST namespace implementation.
For remote object stores with central metadata/basics/catalog services (either commercial or open source),
For remote object stores with central metadata or catalog services (either commercial or open source),
use the REST namespace implementation. It is backed by REST routes
(for example `POST /v1/namespace/{id}/create` and `GET /v1/namespace/{id}/list`) and server-provided table locations.