diff --git a/docs/web/basics/lance.mdx b/docs/web/basics/lance.mdx index 42d474de9..0341ee333 100644 --- a/docs/web/basics/lance.mdx +++ b/docs/web/basics/lance.mdx @@ -34,12 +34,45 @@ those fields together. - **Independent column updates.** Add embeddings or other derived features without rewriting the existing image, video or text columns. +## Opening Lance datasets in Python {#python-packages} + +Any `.lance` dataset directory is a LanceDB table. Pass the parent directory to +`lancedb.connect()` and open the table by its directory stem: + +```python +import lancedb + +db = lancedb.connect("/data") # pass storage_options=... for S3, GCS or Azure +tbl = db.open_table("frames") # opens /data/frames.lance; version=N pins a version +``` + +Use `lancedb` for search, retrieval, data curation, multimodal blob access and +application code. Install `pylance` (`import lance`) only when building query +engines or connectors, calling format-level fragment APIs directly, or +converting plain-scan lazy or legacy blob columns with `to_pandas()` (see +[Multimodal data and blobs](/management/table#multimodal) for `pylance`-free +conversions). + +| `pylance` (`import lance`) | `lancedb` (`import lancedb`) | +| --- | --- | +| `lance.dataset("/data/frames.lance")` | `lancedb.connect("/data").open_table("frames")` | +| `lance.write_dataset(data, "/data/frames.lance")` | `lancedb.connect("/data").create_table("frames", data)` | +| `lance.vector.vec_to_table(mat)` | `db.create_table(name, pa.table({"vector": list(mat)}))` | +| `lance.blob_field("video")` | `lancedb.blob("video")` | +| `ds.take_blobs("video", ids=row_ids)` | `tbl.fetch_blob_files("video", row_ids)` (seekable handles) or `tbl.fetch_blobs("video", row_ids)` (`pa.LargeBinaryArray`) | +| `ds.to_table(columns=cols, filter=expr, limit=n)` | `tbl.search().where(expr).select(cols).limit(n).to_arrow()` | +| `ds.to_batches(batch_size=k)` | `tbl.search().to_batches(k)` | +| `ds.take(offsets)` | `tbl.__getitems__(offsets)` (preserves offset order) or `tbl.take_offsets(offsets).to_arrow()` (unordered) | +| `ds.versions()`, `ds.checkout_version(v)`, `ds.restore()` | `tbl.list_versions()`, `tbl.checkout(v)`, `tbl.restore()` | +| `ds.optimize.compact_files(...)`, `ds.cleanup_old_versions(...)` | `tbl.optimize(compaction_options=..., cleanup_older_than=...)` | + ## Learn more {#next} - [Lance format specification](https://lance.org/format): how the format works. +- [Multimodal data and blobs](/management/table#multimodal): declare `lancedb.blob()` columns and read blobs on demand. - [Table versioning](/management/table#versioning): read and manage table versions. - [Computed columns](/feature-engineering/add-computed-column): add derived features to your data. - [Architecture](/index#architecture): how LanceDB Embedded and Enterprise build on Lance. diff --git a/docs/web/integrations/lerobotdataset.mdx b/docs/web/integrations/lerobotdataset.mdx index 260921034..42abd60b5 100644 --- a/docs/web/integrations/lerobotdataset.mdx +++ b/docs/web/integrations/lerobotdataset.mdx @@ -126,14 +126,14 @@ tbl.add_columns({"jerk_score": jerk_score}) # nothing computed yet tbl.backfill("jerk_score", concurrency=8) # 610,403 rows in 16.7 s ``` -Embeddings work the same way, reading frames straight from the video blobs in the same table: +Embeddings work the same way, reading frames straight from the video blobs in the `videos` table: ```python @udf(data_type=pa.list_(pa.float32(), 768), num_gpus=1, input_columns=["episode_index", "frame_index"]) class EmbedFromBlob: # SigLIP2 loads once per worker def __call__(self, episode_index, frame_index): - blob = self.videos.take_blobs("video_bytes", indices=[vrow])[0] + blob = self.videos.fetch_blob_files("video_bytes", [video_row_id])[0] frame = VideoDecoder(blob).get_frames_at([pos]).data[0] ... ``` diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index 02ec850bb..c86147302 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -63,8 +63,10 @@ uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb ``` -If you also need the optional `lance` Python API, install the `pylance` extra and -include the Lance preview index: +Most applications only need `lancedb`. Install the optional `pylance` extra and +include the Lance preview index only when building query engines or connectors, +calling format-level `lance` APIs directly, or converting plain-scan lazy or +legacy blob columns with `to_pandas()`: ```bash Python (pip) icon="terminal" diff --git a/python/README.md b/python/README.md index 0b640d27c..3db34bc13 100644 --- a/python/README.md +++ b/python/README.md @@ -20,7 +20,7 @@ For those hosts, install the `lancedb-compat` package instead: pip install lancedb-compat ``` -Same Python API (`import lancedb` works as usual). The compat wheel is compiled at the `x86-64-v2` baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use `lance.simd_info()` from Python to verify which tier was selected. +Same Python API (`import lancedb` works as usual). The compat wheel is compiled at the `x86-64-v2` baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon (if `pylance` is installed, `lance.simd_info()` reports which tier was selected). `lancedb` and `lancedb-compat` install to the same `lancedb/` namespace and conflict at install time. Pick one. To switch, `pip uninstall lancedb` first, then `pip install lancedb-compat` (or vice-versa). @@ -40,8 +40,10 @@ Stable releases are created about every 2 weeks. For the latest features and bug pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb ``` -If you also need the optional `lance` Python API, install the `pylance` extra -and include the Lance preview index: +Most applications only need `lancedb`. Install the optional `pylance` extra and +include the Lance preview index only when building query engines or connectors, +calling format-level `lance` APIs directly, or converting plain-scan lazy or +legacy blob columns with `to_pandas()`: ```bash pip install --pre \