lancedb/docs/web/index.mdx

---
title: LanceDB
sidebarTitle: "LanceDB"
description: "Multimodal lakehouse for AI."
icon: "/static/assets/logo/lancedb-icon-gray.svg"
keywords: ["multimodal lakehouse", "training", "feature engineering", "search", "open source", "oss"]
---

**LanceDB** is a [multimodal lakehouse](https://lancedb.com/blog/multimodal-lakehouse/) for AI teams that need
one data layer for curation, feature engineering, search and retrieval, and model training.
It is built on top of [Lance](/basics/lance), an open-source lakehouse format designed for multimodal AI data.

Move from data exploration to model training on one unified platform without needing to manage a
fragmented stack of storage, feature, retrieval, and training systems.

## Train on the right data faster {#train-on-the-right-data-faster}

Search, curate, and train at petabyte scale, with all your multimodal data in one table.

Preparing training data and running experiments slow down when raw data, metadata, embeddings, features, and governance
artifacts live in separate systems. LanceDB keeps them together in one versioned multimodal table, so AI teams spend less
time stitching infrastructure together and more time improving datasets, testing features, and keeping GPUs fed.

## Accelerate the data flywheel for model research {#accelerate-the-data-flywheel-for-model-research}

Keep your AI data in one table, and give each agent-driven experiment its own isolated branch.
Find the failure, fix the data, and retrain.

<Steps>
  <Step title="Find a curated training subset for every failure case">
    Start from one failed grasp or evaluation example. [Search](/search/) multimodal data, metadata, and embeddings
    together, retrieve similar examples, and [curate](/curation) them into a targeted training subset.
  </Step>
  <Step title="Data sandboxes for safe agentic experimentation">
    Don't let an agent silently corrupt your data and ruin a training run. Give each agent its own
    [branch](/tables/branching), a writable fork of the table, to experiment on, and review the changes before
    they reach `main`. See [Run experiments on branches](/agent-branch-experiments).
  </Step>
  <Step title="A single dataset for every modality">
    Keep episodes, frames, metadata, features, and embeddings together in one schema-enforced table, without
    pipelines to keep them in sync. Table [versions](/management/table#versioning) let you trace each training
    run back to the exact data it used.
  </Step>
  <Step title="More hillclimbing, less babysitting preprocessing jobs">
    Define the [computed columns](/feature-engineering/add-computed-column) and
    [materialized views](/feature-engineering/materialized-views) your experiment needs, such as extracted features
    or embeddings, and refresh them as your data changes. On LanceDB Enterprise, column refreshes run as background
    jobs on the cluster instead of in your code.
  </Step>
  <Step title="Higher GPU utilization, faster convergence">
    Keep GPUs fed. [`StreamingDataset`](/training/data-loading) streams multimodal training data into PyTorch with
    globally shuffled, deterministic batches, combining fast random access with high-throughput scans.
  </Step>
</Steps>

Use the same table to curate training data, add derived features, retrieve examples, and feed training jobs that rely on expensive GPUs.
Training workloads can sample, shuffle, and scan projected columns from local storage or object storage, then assemble
GPU-ready batches from a tagged dataset version. Training pipelines need curation, derived features, reproducible
splits, fast random access, and a clean path into frameworks such as PyTorch -- LanceDB keeps all of it connected
through the same table model, whether a workflow uses one table or several related ones.

## Architecture: Embedded and Enterprise {#architecture}

LanceDB ships as two offerings on the same [Lance](/basics/lance) format and table model, which stores table data,
versions, and indexes in open, columnar files. What changes between them is how storage and compute are organized
around that format. Start locally with Embedded, then move to Enterprise when your team needs distributed scale,
managed infrastructure, private deployment, or higher-throughput curation, feature engineering, search and
retrieval, and training workflows.

<Columns cols={2}>
  <Frame caption="LanceDB Embedded: the LanceDB SDK and the Lance format run in your application, reading and writing tables on local disk or in object storage.">
    <img src="/static/assets/images/overview/lancedb-embedded.png" alt="LanceDB Embedded architecture" />
  </Frame>
  <Frame caption="LanceDB Enterprise: the LanceDB SDK connects to the Enterprise service, where a query fleet, executor and cache fleets, the realtime engine, and a jobs fleet run on top of the Lance format and object storage.">
    <img src="/static/assets/images/overview/lancedb-distributed.png" alt="LanceDB Enterprise architecture" />
  </Frame>
</Columns>

### 1\. LanceDB Embedded {#1-lancedb-oss}

**LanceDB Embedded** is an open-source, embedded retrieval library, with client SDKs in Python, TypeScript and
Rust -- the application itself plans queries, builds indexes, and runs background work, with no separate database
server to deploy. It reads and writes tables on local disk or in object storage from inside your application
process. This fits local retrieval applications, notebooks, development, and single-node data pipelines, where
your application controls its own compute resources and [table maintenance](/maintenance/compaction).

### 2\. LanceDB Enterprise {#2-lancedb-enterprise}

**LanceDB Enterprise** is an exabyte-scale, distributed **multimodal lakehouse** service built on the
same table abstraction as Embedded, for search, curation, feature engineering, and high-throughput training data
access. Applications connect through a LanceDB SDK while the service plans and runs work across a fleet: query
serving, read execution and caching, indexing, and background maintenance scale as separate components, with
durable table data and index artifacts kept in object storage, independently of the machines serving requests.
Caches accelerate reads; they are never the durable source of truth. With multiple serving replicas, Enterprise
routes traffic to surviving nodes when one fails; losing a cache node temporarily raises latency while its
replacement warms, rather than causing an outage. See [Distributed search](/enterprise/distributed-search),
[distributed indexing](/enterprise/distributed-indexing), and the other architecture guides linked from the
comparison table below for how work divides across machines.

Teams choose Enterprise to:

- **Get new capabilities first** -- SQL, Functions, materialized views, and most other features documented
  throughout these docs land in Enterprise first, often well before they reach Embedded, if they ever do.
- **Performance** -- cache frequently read table data near compute with the [Data Page Cache](/enterprise/data-page-cache),
  and let [adaptive execution](/enterprise/adaptive-execution) size refresh and curation workers to the job instead
  of guessing once.
- **Scale** -- add query-serving capacity by spreading a table across more [replica groups](/enterprise/horizontal-scaling)
  as traffic grows, instead of upgrading one machine.
- **Fault tolerance** -- [replication](/enterprise/horizontal-scaling) keeps a table serving when a node fails, so
  you lose latency to a warm-up, not an outage.
- **Multicloud deployment** -- bring LanceDB into your own cloud account, in the region your
  [GPU capacity](/curation/gpu-function-execution) already lives, instead of moving data to it.
- **Simplify operations** -- [Intelligent Auto-Maintenance](/enterprise/intelligent-auto-maintenance) runs
  compaction, cleanup, and indexing in the background, with [OpenTelemetry](/deployment/opentelemetry) metrics and
  traces instead of guesswork.

[Reach out to us](https://www.lancedb.com/contact) to set up LanceDB Enterprise in your organization. Sharing a
format does not mean an Enterprise deployment automatically opens an existing Embedded database -- plan how to
move or connect your data.

### Compare features {#compare}

| Area | Feature | Embedded | Enterprise |
| --- | --- | :-: | :-: |
| **Data Ingestion** | [Insert data](/ingestion/insert) | ✅ | ✅ |
|  | [Merge](/ingestion/merge-insert) | ✅ | ✅ |
|  | [RTE Write](/ingestion/realtime-write) | ✅ | ✅ |
|  | [Distributed RTE Write](/enterprise/distributed-real-time-write) | ❌ | ✅ |
|  | [Attach Database](/ingestion/migrate-from-embedded) | ❌ | ✅ |
|  | [Import Datasets](/ingestion/import) | ❌ | ✅ |
|  | [Distributed Write](/enterprise/distributed-write) | ❌ | ✅ |
| **Semantic Search** | [Vector search](/search/vector-search) | ✅ | ✅ |
|  | [Full-text search](/search/full-text-search) | ✅ | ✅ |
|  | [Hybrid search](/search/hybrid-search) | ✅ | ✅ |
|  | [Filtering](/search/filtering) | ✅ | ✅ |
|  | [Reranking](/reranking) | ✅ | ✅ |
|  | [Distributed search](/enterprise/distributed-search) | ❌ | ✅ |
|  | [RTE Read](/search/realtime-read) | ✅ | ✅ |
|  | [Distributed RTE Read](/enterprise/distributed-real-time-read) | ❌ | ✅ |
|  | [Cache Prewarm](/enterprise/cache-prewarm) | ❌ | ✅ |
| **Data Curation** | [Inspect Datasets](/curation/inspect-datasets) | ❌ | ✅ |
|  | [Statistical Sampling](/curation/statistical-sampling) | ❌ | ✅ |
|  | [Selective Sampling](/curation/selective-sampling) | ✅ | ✅ |
|  | [Aggregation and profiling](/curation/aggregate) | ❌ | ✅ |
|  | [Joins](/curation/join) | ❌ | ✅ |
|  | [Blob Late Materialization](/curation/blob-late-materialization) | ❌ | ✅ |
|  | [Ranking and deduplication](/curation/windowing) | ❌ | ✅ |
|  | [Near-duplicate detection](/curation/dedupe) | ❌ | ✅ |
|  | [Chunking](/curation/chunking) | ❌ | ✅ |
|  | [Changes since a version](/curation/changes) | ❌ | ✅ |
|  | [GPU Function Execution](/curation/gpu-function-execution) | ❌ | ✅ |
|  | [Slurm Scheduling](/curation/slurm-scheduling) | ❌ | ✅ |
| **Feature Engineering** | [Computed columns](/feature-engineering/add-computed-column) | ❌ | ✅ |
|  | [Incremental Refresh](/enterprise/incremental-refresh) | ❌ | ✅ |
|  | [Auto Refresh](/enterprise/auto-refresh) | ❌ | ✅ |
|  | [Checkpointing](/enterprise/checkpointing) | ❌ | ✅ |
|  | [Adaptive Execution](/enterprise/adaptive-execution) | ❌ | ✅ |
|  | [GPU Refresh Execution](/enterprise/gpu-refresh-execution) | ❌ | ✅ |
|  | [Slurm Scheduling](/enterprise/slurm-refresh-scheduling) | ❌ | ✅ |
|  | [Refresh Materialized View](/feature-engineering/refresh-materialized-view) | ❌ | ✅ |
|  | [Embedding registry](/embedding) | ✅ | ✅ |
| **Training** | [PyTorch data loader](/training/data-loading) | ✅ | ✅ |
|  | [Training Cache](/training/training-cache) | ❌ | ✅ |
|  | [Permutations](/training/permutation) | ✅ | ✅ |
|  | [Distributed training](/training/distributed-training) | ✅ | ✅ |
|  | [Observability](/training/observability) | ✅ | ✅ |
|  | [Checkpointing](/training/checkpointing) | ✅ | ✅ |
|  | [Resuming](/training/resuming) | ✅ | ✅ |
| **Management** | [Catalog](/management/catalog) | ❌ | ✅ |
|  | [Database](/management/database) | ✅ | ✅ |
|  | [Namespace](/management/namespace) | ✅ | ✅ |
|  | [Table](/management/table) | ✅ | ✅ |
|  | [Branches](/tables/branching) | ✅ | ✅ |
|  | [View](/management/view) | ❌ | ✅ |
|  | [Materialized view](/management/materialized-view) | ❌ | ✅ |
|  | [Secret](/management/secret) | ❌ | ✅ |
|  | [Function](/management/function) | ❌ | ✅ |
|  | [Storage integration](/management/storage-integration) | ❌ | ✅ |
| **Maintenance** | [Intelligent Auto-Maintenance](/enterprise/intelligent-auto-maintenance) | ❌ | ✅ |
|  | [Vector Indexing](/indexing/vector-index) | ✅ | ✅ |
|  | [FTS Indexing](/indexing/fts-index) | ✅ | ✅ |
|  | [Scalar Indexing](/indexing/scalar-index) | ✅ | ✅ |
|  | [Incremental Indexing](/indexing/incremental-indexing) | ✅ | ✅ |
|  | [Distributed Indexing](/enterprise/distributed-indexing) | ❌ | ✅ |
|  | [GPU Indexing](/indexing/gpu-indexing) | ✅ | ✅ |
|  | [Compaction](/maintenance/compaction) | ✅ | ✅ |
|  | [RTE Compaction](/enterprise/realtime-engine-compaction) | ❌ | ✅ |
|  | [Garbage Collection](/maintenance/cleanup) | ✅ | ✅ |
|  | [Distributed compaction](/enterprise/distributed-compaction) | ❌ | ✅ |
|  | [Advanced Conflict Resolution](/enterprise/advanced-conflict-resolution) | ❌ | ✅ |
| **Security** | [Authentication](/enterprise/authentication) | ❌ | ✅ |
|  | [Authorization](/governance/authorization) | ❌ | ✅ |
|  | [Audit Log](/governance/audit-log) | ❌ | ✅ |
| **Deployment** | [Horizontal Scaling](/enterprise/horizontal-scaling) | ❌ | ✅ |
|  | [OpenTelemetry](/deployment/opentelemetry) | ✅ | ✅ |
|  | [Kubernetes](/enterprise/deployment/kubernetes) | ❌ | ✅ |

## Projects using LanceDB {#projects-using-lancedb}

### Training {#training}

- [stable-worldmodel](https://github.com/galilai-group/stable-worldmodel) -- a platform for reproducible world-model
  research built on a LanceDB data layer, reporting 3-4x faster data loading on Push-T versus HDF5 / MP4 at a
  fraction of the disk footprint.
- [le-wm](https://github.com/lucas-maes/le-wm) -- a joint-embedding predictive world model from pixels, trained on
  the stable-worldmodel platform and its LanceDB data layer.
- [lerobot-lancedb](https://github.com/lancedb/lerobot-lancedb) -- a drop-in LanceDB backend for HuggingFace
  LeRobot datasets with faster loading across robotics datasets.

### Agentic applications {#agentic-applications}

- [OpenClaw LanceDB Plugin](https://docs.openclaw.ai/plugins/memory-lancedb) -- stores an agent's long-term memory
  in LanceDB, automatically recalling relevant memories before a model turn and capturing new facts afterward.
- [Cognee](https://github.com/topoteretes/cognee) -- an open-source AI memory platform that gives agents persistent
  memory across sessions, using LanceDB as one of its vector store backends.
- [CrewAI](https://github.com/crewAIInc/crewAI) -- a multi-agent framework that
  [rebuilt its agent memory on LanceDB](https://www.lancedb.com/blog/crewai-rebuilt-agent-memory-on-lancedb).

Across these projects, LanceDB and Lance provide the multimodal data layer that keeps raw observations, annotations,
features, and training or memory access patterns in one format instead of scattering them across task-specific
stores.

## Start with your workload {#start-with-your-workload}

<CardGroup cols={2}>
  <Card title="Ingest your data" icon="inbox" href="/ingestion">
    Create tables, append records, and update or delete rows as your data changes.
  </Card>
  <Card title="Build search and retrieval" icon="search" href="/search/">
    Use vector search, full-text search, hybrid search, reranking, filtering, and SQL.
  </Card>
  <Card title="Curate training data" icon="filter" href="/curation">
    Profile, sample, chunk, join, and deduplicate records before training on them.
  </Card>
  <Card title="Engineer features in place" icon="layer-group" href="/feature-engineering">
    Derive computed columns and materialized views without moving data elsewhere.
  </Card>
  <Card title="Load data into PyTorch" icon="boxes-stacked" href="/training/data-loading">
    Use LanceDB tables and permutations for projected, shuffled, random-access training reads.
  </Card>
  <Card title="Manage catalog objects" icon="folder-tree" href="/management">
    Create, list, and drop tables, namespaces, and views from Python, TypeScript, or SQL.
  </Card>
</CardGroup>