mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-30 15:26:52 +00:00
Will JonesandClaude Opus 5.5
4d7a64712f
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
129 lines
6.7 KiB
Plaintext
129 lines
6.7 KiB
Plaintext
---
|
|
title: "Architecture"
|
|
sidebarTitle: "Architecture"
|
|
description: "How LanceDB Enterprise separates query serving, background work, and storage, and how it compares with LanceDB Embedded."
|
|
icon: cubes
|
|
keywords: ["architecture", "embedded", "distributed", "object storage", "compute", "deployment"]
|
|
---
|
|
|
|
LanceDB Embedded is an open-source library for working with Lance tables. It runs
|
|
in your application's process; your data can live on local disk or in object
|
|
storage. LanceDB Enterprise is a multimodal lakehouse for data curation, feature
|
|
engineering, search, and training data access. LanceDB Enterprise runs LanceDB
|
|
Distributed: query execution, indexing and maintenance run on the Enterprise
|
|
service, outside your application. Its architecture separates query serving,
|
|
read execution, caching, and background work. Both use the open
|
|
[Lance](/basics/lance) format.
|
|
|
|
Teams can curate datasets, derive features, search multimodal records, and
|
|
read the resulting tables into training frameworks. Enterprise separates the
|
|
resources that serve shared data from those that build indexes and run
|
|
background work, so teams can size them for these different workloads.
|
|
|
|
|
|
|
|
The diagram shows the main compute and storage boundaries, not every request
|
|
path. The control plane governs the Enterprise service.
|
|
|
|
## A shared format, different storage choices {#storage}
|
|
|
|
Lance stores table data, versions, and indexes. LanceDB Embedded can read and write tables on
|
|
local disk or in object storage. Enterprise keeps durable table data and index
|
|
artifacts in object storage, independently of the machines serving requests.
|
|
Its caches accelerate reads; they are not the durable source of truth.
|
|
|
|
Sharing a format does not mean that an Enterprise deployment automatically
|
|
opens an existing LanceDB Embedded database. Plan how to move or connect your data and
|
|
configure access for your deployment.
|
|
|
|
## LanceDB Embedded: a library inside your application {#embedded}
|
|
|
|
With an embedded connection, the LanceDB library plans and executes queries,
|
|
builds indexes, and updates tables inside your process. There is no separate
|
|
database server to deploy.
|
|
|
|
This fits local retrieval applications, notebooks, development, and
|
|
single-node data pipelines. Your application controls the compute resources and
|
|
when to run [table maintenance](/maintenance/compaction). You also manage access
|
|
to the local files or object storage that hold your tables.
|
|
|
|
## LanceDB Enterprise: distributed execution for shared workloads {#distributed}
|
|
|
|
A LanceDB SDK connects to LanceDB Enterprise, and the service runs operations
|
|
with LanceDB Distributed. The service handles
|
|
requests across a fleet, with components sized for different kinds of work:
|
|
|
|
- **Query serving** receives requests, resolves tables, and plans operations.
|
|
Add serving capacity as more applications and users depend on your data.
|
|
- **Read execution and caching** use separately provisioned resources. Executors
|
|
cache table data to reduce repeated reads from object storage.
|
|
- **Indexing and background workers** build indexes and perform maintenance
|
|
outside the query request path. Size this work separately from query serving
|
|
as your data grows.
|
|
|
|
The control plane manages configuration, identity, policy, and cluster
|
|
lifecycle. It governs the service rather than executing table queries.
|
|
[Authorization](/governance/authorization) lets teams control access to catalog
|
|
objects when enabled and configured by the administrator. Deployment settings govern background
|
|
[compaction](/maintenance/compaction) and
|
|
[cleanup](/maintenance/cleanup), including retention.
|
|
|
|
These boundaries support different parts of an AI data workflow.
|
|
[SQL curation](/curation) requires a deployment with SQL enabled.
|
|
[Feature engineering](/feature-engineering) can produce derived columns and
|
|
tables; computed columns and materialized views require a deployment that
|
|
enables them. [Geneva](/geneva) runs distributed Python feature backfills while
|
|
its capabilities continue to transition to LanceDB APIs.
|
|
[Training data loaders](/training/data-loading) read tables into model training
|
|
frameworks in both offerings. LanceDB provides data access for training; the
|
|
training framework runs the model.
|
|
|
|
Size and monitor each tier for your workload. Read the
|
|
[Enterprise architecture](/enterprise/architecture) for the component details
|
|
and request flows.
|
|
|
|
## Application changes for Enterprise {#differences}
|
|
|
|
Core operations remain familiar: create a table, add data, and run a search.
|
|
With LanceDB Embedded, the application executes those operations. With Enterprise, the SDK
|
|
sends requests to the service, and the service manages execution and configured
|
|
background maintenance.
|
|
|
|
To use Enterprise, configure the cluster endpoint, database URI and credentials,
|
|
and make your data available in the Enterprise database. Review SDK methods
|
|
that behave differently with LanceDB Distributed, including result retrieval and
|
|
background jobs. [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise)
|
|
describes these application changes. If you plan to use materialized views, check the
|
|
[stable row ID requirements](/basics/limits#schema) before creating source tables.
|
|
|
|
## Choose for the workload you need to run {#choosing}
|
|
|
|
Choose LanceDB Embedded when you want an embedded library and can manage its resources and
|
|
maintenance in your application.
|
|
|
|
Choose Enterprise when multiple applications or teams need a shared multimodal
|
|
lakehouse for curation, feature engineering, search, and training data access.
|
|
It also fits workloads whose query and indexing demand grow at different rates,
|
|
or that need configured service-level access controls and background operations.
|
|
You can begin with Enterprise.
|
|
|
|
[Talk to us about your workload](https://www.lancedb.com/contact) to plan capacity,
|
|
deployment options, and the path from your existing data to Enterprise.
|
|
|
|
## Where to go next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Enterprise architecture" icon="cubes" href="/enterprise/architecture">
|
|
Query serving, execution, and background work in detail.
|
|
</Card>
|
|
<Card title="Connect to Enterprise" icon="plug" href="/basics/connect#enterprise">
|
|
Connect an application to your deployment.
|
|
</Card>
|
|
<Card title="Guarantees" icon="shield-check" href="/basics/guarantees">
|
|
What a write promises, and what a reader sees.
|
|
</Card>
|
|
<Card title="Lance format" icon="layer-group" href="/basics/lance">
|
|
The open format underneath both offerings.
|
|
</Card>
|
|
</CardGroup>
|