mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
Xuanwo
1e5b6deee3
## Summary - Let local plain queries over row-addressable legacy v1 and v2 blob columns use `blob_mode="bytes"` without importing `pylance` through pandas, batch, or Arrow conversion. - Preserve aliases, filters, ordering, limits, null and empty payloads, empty result schemas, and the caller's `_rowid` choice. - Build the hidden row-id query separately so reusing an async builder for lazy reads or non-blob projections cannot retain an automatic `_rowid`. - Keep byte reads closed for blobs inside lists, remote or namespace pushdown, and query shapes whose result rows cannot be matched back to the source. Blob v2 lazy auto-row-id behavior is unchanged. Fixes #4645 ## Validation - 26 focused Python tests passed with the test subprocess blocking all `lance` imports. The persisted fixture writes and reopens a storage-v2.1 table, then covers sync and async pandas, batches, and Arrow conversion plus async builder reuse. - The existing unsupported vector and hybrid reranker regressions now assert the current eligibility diagnostic, and the legacy table-level Arrow regression now asserts successful bytes, schema, values, and hidden row IDs. Both focused tests passed. - The runtime receipt records no installed `pylance` distribution and no importable `lance` module. The Python-only change was verified with an existing compatible 0.42.0b9 extension; the relevant query and fetch APIs are unchanged between that source and this 0.42.0b10 base. - Ruff format and lint pass for every changed Python file.
80 lines
3.9 KiB
Plaintext
80 lines
3.9 KiB
Plaintext
---
|
|
title: "Lance format"
|
|
sidebarTitle: "Lance Format"
|
|
description: "The open format for multimodal data behind LanceDB."
|
|
icon: "/static/assets/logo/lance-logo-gray.png"
|
|
keywords: ["lance", "format", "multimodal", "versioning", "column evolution"]
|
|
---
|
|
|
|
## Lance format overview {#standard}
|
|
|
|
[Lance](https://lance.org/) is an open-source, columnar format designed for
|
|
multimodal AI. It defines how data is stored and organized into versioned tables.
|
|
|
|
<span id="embedded" />
|
|
<span id="enterprise" />
|
|
|
|
LanceDB Embedded and LanceDB Enterprise both use Lance as their storage foundation.
|
|
The format is developed as an independent open-source project, with its own
|
|
specification and tools.
|
|
|
|
## Multimodal data {#storage}
|
|
|
|
A Lance table can hold images, video, audio, text, embeddings and structured
|
|
metadata together. For example, a video dataset can keep each clip alongside
|
|
its caption, labels and embeddings, so applications can search and work with
|
|
those fields together.
|
|
|
|
## Data access, versioning and column updates {#capabilities}
|
|
|
|
- **Efficient data access.** Read individual rows for retrieval and training,
|
|
or scan columns for analytics.
|
|
- **Versioned data.** Read an earlier table version to reproduce an experiment
|
|
or inspect previous results, while that version is retained.
|
|
- **Independent column updates.** Add embeddings or other derived features
|
|
without rewriting the existing image, video or text columns.
|
|
|
|
## Opening Lance datasets in Python {#python-packages}
|
|
|
|
Any `.lance` dataset directory is a LanceDB table. Pass the parent directory to
|
|
`lancedb.connect()` and open the table by its directory stem:
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
db = lancedb.connect("/data") # pass storage_options=... for S3, GCS or Azure
|
|
tbl = db.open_table("frames") # opens /data/frames.lance; version=N pins a version
|
|
```
|
|
|
|
Use `lancedb` for search, retrieval, data curation, multimodal blob access and
|
|
application code. Install `pylance` (`import lance`) only when building query
|
|
engines or connectors, calling format-level fragment APIs directly, or
|
|
converting plain-scan lazy blob handles with `to_pandas()`. Legacy plain-scan
|
|
byte conversions use LanceDB's native path without `pylance` (see
|
|
[Multimodal data and blobs](/management/table#multimodal) for conversion
|
|
boundaries).
|
|
|
|
| `pylance` (`import lance`) | `lancedb` (`import lancedb`) |
|
|
| --- | --- |
|
|
| `lance.dataset("/data/frames.lance")` | `lancedb.connect("/data").open_table("frames")` |
|
|
| `lance.write_dataset(data, "/data/frames.lance")` | `lancedb.connect("/data").create_table("frames", data)` |
|
|
| `lance.vector.vec_to_table(mat)` | `db.create_table(name, pa.table({"vector": list(mat)}))` |
|
|
| `lance.blob_field("video")` | `lancedb.blob("video")` |
|
|
| `ds.take_blobs("video", ids=row_ids)` | `tbl.fetch_blob_files("video", row_ids)` (seekable handles) or `tbl.fetch_blobs("video", row_ids)` (`pa.LargeBinaryArray`) |
|
|
| `ds.to_table(columns=cols, filter=expr, limit=n)` | `tbl.search().where(expr).select(cols).limit(n).to_arrow()` |
|
|
| `ds.to_batches(batch_size=k)` | `tbl.search().to_batches(k)` |
|
|
| `ds.take(offsets)` | `tbl.__getitems__(offsets)` (preserves offset order) or `tbl.take_offsets(offsets).to_arrow()` (unordered) |
|
|
| `ds.versions()`, `ds.checkout_version(v)`, `ds.restore()` | `tbl.list_versions()`, `tbl.checkout(v)`, `tbl.restore()` |
|
|
| `ds.optimize.compact_files(...)`, `ds.cleanup_old_versions(...)` | `tbl.optimize(compaction_options=..., cleanup_older_than=...)` |
|
|
|
|
## Learn more {#next}
|
|
|
|
<span id="versioning" />
|
|
<span id="evolution" />
|
|
|
|
- [Lance format specification](https://lance.org/format): how the format works.
|
|
- [Multimodal data and blobs](/management/table#multimodal): declare `lancedb.blob()` columns and read blobs on demand.
|
|
- [Table versioning](/management/table#versioning): read and manage table versions.
|
|
- [Computed columns](/feature-engineering/add-computed-column): add derived features to your data.
|
|
- [Architecture](/index#architecture): how LanceDB Embedded and Enterprise build on Lance.
|