mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Xuanwo
28ba34998a
Replace the remaining `pylance` (`take_blobs`) snippet in the LeRobot integration guide with `Table.fetch_blob_files`, document how to open `.lance` datasets directly with `lancedb` alongside a `pylance`-to-`lancedb` call mapping on `/basics/lance`, and clarify in installation docs that the optional `pylance` extra is only needed for engine/connector authors or format-level fragment APIs.
79 lines
3.8 KiB
Plaintext
79 lines
3.8 KiB
Plaintext
---
|
|
title: "Lance format"
|
|
sidebarTitle: "Lance Format"
|
|
description: "The open format for multimodal data behind LanceDB."
|
|
icon: "/static/assets/logo/lance-logo-gray.png"
|
|
keywords: ["lance", "format", "multimodal", "versioning", "column evolution"]
|
|
---
|
|
|
|
## Lance format overview {#standard}
|
|
|
|
[Lance](https://lance.org/) is an open-source, columnar format designed for
|
|
multimodal AI. It defines how data is stored and organized into versioned tables.
|
|
|
|
<span id="embedded" />
|
|
<span id="enterprise" />
|
|
|
|
LanceDB Embedded and LanceDB Enterprise both use Lance as their storage foundation.
|
|
The format is developed as an independent open-source project, with its own
|
|
specification and tools.
|
|
|
|
## Multimodal data {#storage}
|
|
|
|
A Lance table can hold images, video, audio, text, embeddings and structured
|
|
metadata together. For example, a video dataset can keep each clip alongside
|
|
its caption, labels and embeddings, so applications can search and work with
|
|
those fields together.
|
|
|
|
## Data access, versioning and column updates {#capabilities}
|
|
|
|
- **Efficient data access.** Read individual rows for retrieval and training,
|
|
or scan columns for analytics.
|
|
- **Versioned data.** Read an earlier table version to reproduce an experiment
|
|
or inspect previous results, while that version is retained.
|
|
- **Independent column updates.** Add embeddings or other derived features
|
|
without rewriting the existing image, video or text columns.
|
|
|
|
## Opening Lance datasets in Python {#python-packages}
|
|
|
|
Any `.lance` dataset directory is a LanceDB table. Pass the parent directory to
|
|
`lancedb.connect()` and open the table by its directory stem:
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
db = lancedb.connect("/data") # pass storage_options=... for S3, GCS or Azure
|
|
tbl = db.open_table("frames") # opens /data/frames.lance; version=N pins a version
|
|
```
|
|
|
|
Use `lancedb` for search, retrieval, data curation, multimodal blob access and
|
|
application code. Install `pylance` (`import lance`) only when building query
|
|
engines or connectors, calling format-level fragment APIs directly, or
|
|
converting plain-scan lazy or legacy blob columns with `to_pandas()` (see
|
|
[Multimodal data and blobs](/management/table#multimodal) for `pylance`-free
|
|
conversions).
|
|
|
|
| `pylance` (`import lance`) | `lancedb` (`import lancedb`) |
|
|
| --- | --- |
|
|
| `lance.dataset("/data/frames.lance")` | `lancedb.connect("/data").open_table("frames")` |
|
|
| `lance.write_dataset(data, "/data/frames.lance")` | `lancedb.connect("/data").create_table("frames", data)` |
|
|
| `lance.vector.vec_to_table(mat)` | `db.create_table(name, pa.table({"vector": list(mat)}))` |
|
|
| `lance.blob_field("video")` | `lancedb.blob("video")` |
|
|
| `ds.take_blobs("video", ids=row_ids)` | `tbl.fetch_blob_files("video", row_ids)` (seekable handles) or `tbl.fetch_blobs("video", row_ids)` (`pa.LargeBinaryArray`) |
|
|
| `ds.to_table(columns=cols, filter=expr, limit=n)` | `tbl.search().where(expr).select(cols).limit(n).to_arrow()` |
|
|
| `ds.to_batches(batch_size=k)` | `tbl.search().to_batches(k)` |
|
|
| `ds.take(offsets)` | `tbl.__getitems__(offsets)` (preserves offset order) or `tbl.take_offsets(offsets).to_arrow()` (unordered) |
|
|
| `ds.versions()`, `ds.checkout_version(v)`, `ds.restore()` | `tbl.list_versions()`, `tbl.checkout(v)`, `tbl.restore()` |
|
|
| `ds.optimize.compact_files(...)`, `ds.cleanup_old_versions(...)` | `tbl.optimize(compaction_options=..., cleanup_older_than=...)` |
|
|
|
|
## Learn more {#next}
|
|
|
|
<span id="versioning" />
|
|
<span id="evolution" />
|
|
|
|
- [Lance format specification](https://lance.org/format): how the format works.
|
|
- [Multimodal data and blobs](/management/table#multimodal): declare `lancedb.blob()` columns and read blobs on demand.
|
|
- [Table versioning](/management/table#versioning): read and manage table versions.
|
|
- [Computed columns](/feature-engineering/add-computed-column): add derived features to your data.
|
|
- [Architecture](/index#architecture): how LanceDB Embedded and Enterprise build on Lance.
|