mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 07:36:50 +00:00
Shortens the docs landing page and organizes its overview around
database features, with the multimodal data model, ML workflow, SDKs,
and PyTorch loader explained in prose. Keeps versioning visible,
branching in draft, and the existing `{#architecture}` anchor.
Validation: added documentation links and anchors verified; `git diff
--check` passes. `mint validate` reports the same four pre-existing
missing snippet-export warnings in the reranking and vector-search
pages, also reproduced on the unchanged revision.
Closes
https://linear.app/lancedb/issue/DOC-80/strengthen-what-is-lancedb
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
80 lines
3.9 KiB
Plaintext
80 lines
3.9 KiB
Plaintext
---
|
|
title: "Lance format"
|
|
sidebarTitle: "Lance Format"
|
|
description: "The open format for multimodal data behind LanceDB."
|
|
icon: "/static/assets/logo/lance-logo-gray.png"
|
|
keywords: ["lance", "format", "multimodal", "versioning", "column evolution"]
|
|
---
|
|
|
|
## Lance format overview {#standard}
|
|
|
|
[Lance](https://lance.org/) is an open-source, columnar format designed for
|
|
multimodal AI. It defines how data is stored and organized into versioned tables.
|
|
|
|
<span id="embedded" />
|
|
<span id="enterprise" />
|
|
|
|
LanceDB Embedded and LanceDB Enterprise both use Lance as their storage foundation.
|
|
The format is developed as an independent open-source project, with its own
|
|
specification and tools.
|
|
|
|
## Multimodal data {#storage}
|
|
|
|
A Lance table can hold images, video, audio, text, embeddings and structured
|
|
metadata together. For example, a video dataset can keep each clip alongside
|
|
its caption, labels and embeddings, so applications can search and work with
|
|
those fields together.
|
|
|
|
## Data access, versioning and column updates {#capabilities}
|
|
|
|
- **Efficient data access.** Read individual rows for retrieval and training,
|
|
or scan columns for analytics.
|
|
- **Versioned data.** Read an earlier table version to reproduce an experiment
|
|
or inspect previous results, while that version is retained.
|
|
- **Independent column updates.** Add embeddings or other derived features
|
|
without rewriting the existing image, video or text columns.
|
|
|
|
## Opening Lance datasets in Python {#python-packages}
|
|
|
|
Any `.lance` dataset directory is a LanceDB table. Pass the parent directory to
|
|
`lancedb.connect()` and open the table by its directory stem:
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
db = lancedb.connect("/data") # pass storage_options=... for S3, GCS or Azure
|
|
tbl = db.open_table("frames") # opens /data/frames.lance; version=N pins a version
|
|
```
|
|
|
|
Use `lancedb` for search, retrieval, data curation, multimodal blob access and
|
|
application code. Install `pylance` (`import lance`) only when building query
|
|
engines or connectors, calling format-level fragment APIs directly, or
|
|
converting plain-scan lazy blob handles with `to_pandas()`. Legacy plain-scan
|
|
byte conversions use LanceDB's native path without `pylance` (see
|
|
[Multimodal data and blobs](/management/table#multimodal) for conversion
|
|
boundaries).
|
|
|
|
| `pylance` (`import lance`) | `lancedb` (`import lancedb`) |
|
|
| --- | --- |
|
|
| `lance.dataset("/data/frames.lance")` | `lancedb.connect("/data").open_table("frames")` |
|
|
| `lance.write_dataset(data, "/data/frames.lance")` | `lancedb.connect("/data").create_table("frames", data)` |
|
|
| `lance.vector.vec_to_table(mat)` | `db.create_table(name, pa.table({"vector": list(mat)}))` |
|
|
| `lance.blob_field("video")` | `lancedb.blob("video")` |
|
|
| `ds.take_blobs("video", ids=row_ids)` | `tbl.fetch_blob_files("video", row_ids)` (seekable handles) or `tbl.fetch_blobs("video", row_ids)` (`pa.LargeBinaryArray`) |
|
|
| `ds.to_table(columns=cols, filter=expr, limit=n)` | `tbl.search().where(expr).select(cols).limit(n).to_arrow()` |
|
|
| `ds.to_batches(batch_size=k)` | `tbl.search().to_batches(k)` |
|
|
| `ds.take(offsets)` | `tbl.__getitems__(offsets)` (preserves offset order) or `tbl.take_offsets(offsets).to_arrow()` (unordered) |
|
|
| `ds.versions()`, `ds.checkout_version(v)`, `ds.restore()` | `tbl.list_versions()`, `tbl.checkout(v)`, `tbl.restore()` |
|
|
| `ds.optimize.compact_files(...)`, `ds.cleanup_old_versions(...)` | `tbl.optimize(compaction_options=..., cleanup_older_than=...)` |
|
|
|
|
## Learn more {#next}
|
|
|
|
<span id="versioning" />
|
|
<span id="evolution" />
|
|
|
|
- [Lance format specification](https://lance.org/format): how the format works.
|
|
- [Multimodal data and blobs](/management/table#multimodal): declare `lancedb.blob()` columns and read blobs on demand.
|
|
- [Table versioning](/management/table#versioning): read and manage table versions.
|
|
- [Computed columns](/feature-engineering/add-computed-column): add derived features to your data.
|
|
- [Embedded vs Enterprise](/basics/embedded-vs-enterprise): how LanceDB Embedded and Enterprise build on Lance.
|