mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
docs: replace pylance calls with native lancedb equivalents (#4811)
Replace the remaining `pylance` (`take_blobs`) snippet in the LeRobot integration guide with `Table.fetch_blob_files`, document how to open `.lance` datasets directly with `lancedb` alongside a `pylance`-to-`lancedb` call mapping on `/basics/lance`, and clarify in installation docs that the optional `pylance` extra is only needed for engine/connector authors or format-level fragment APIs.
This commit is contained in:
@@ -34,12 +34,45 @@ those fields together.
|
||||
- **Independent column updates.** Add embeddings or other derived features
|
||||
without rewriting the existing image, video or text columns.
|
||||
|
||||
## Opening Lance datasets in Python {#python-packages}
|
||||
|
||||
Any `.lance` dataset directory is a LanceDB table. Pass the parent directory to
|
||||
`lancedb.connect()` and open the table by its directory stem:
|
||||
|
||||
```python
|
||||
import lancedb
|
||||
|
||||
db = lancedb.connect("/data") # pass storage_options=... for S3, GCS or Azure
|
||||
tbl = db.open_table("frames") # opens /data/frames.lance; version=N pins a version
|
||||
```
|
||||
|
||||
Use `lancedb` for search, retrieval, data curation, multimodal blob access and
|
||||
application code. Install `pylance` (`import lance`) only when building query
|
||||
engines or connectors, calling format-level fragment APIs directly, or
|
||||
converting plain-scan lazy or legacy blob columns with `to_pandas()` (see
|
||||
[Multimodal data and blobs](/management/table#multimodal) for `pylance`-free
|
||||
conversions).
|
||||
|
||||
| `pylance` (`import lance`) | `lancedb` (`import lancedb`) |
|
||||
| --- | --- |
|
||||
| `lance.dataset("/data/frames.lance")` | `lancedb.connect("/data").open_table("frames")` |
|
||||
| `lance.write_dataset(data, "/data/frames.lance")` | `lancedb.connect("/data").create_table("frames", data)` |
|
||||
| `lance.vector.vec_to_table(mat)` | `db.create_table(name, pa.table({"vector": list(mat)}))` |
|
||||
| `lance.blob_field("video")` | `lancedb.blob("video")` |
|
||||
| `ds.take_blobs("video", ids=row_ids)` | `tbl.fetch_blob_files("video", row_ids)` (seekable handles) or `tbl.fetch_blobs("video", row_ids)` (`pa.LargeBinaryArray`) |
|
||||
| `ds.to_table(columns=cols, filter=expr, limit=n)` | `tbl.search().where(expr).select(cols).limit(n).to_arrow()` |
|
||||
| `ds.to_batches(batch_size=k)` | `tbl.search().to_batches(k)` |
|
||||
| `ds.take(offsets)` | `tbl.__getitems__(offsets)` (preserves offset order) or `tbl.take_offsets(offsets).to_arrow()` (unordered) |
|
||||
| `ds.versions()`, `ds.checkout_version(v)`, `ds.restore()` | `tbl.list_versions()`, `tbl.checkout(v)`, `tbl.restore()` |
|
||||
| `ds.optimize.compact_files(...)`, `ds.cleanup_old_versions(...)` | `tbl.optimize(compaction_options=..., cleanup_older_than=...)` |
|
||||
|
||||
## Learn more {#next}
|
||||
|
||||
<span id="versioning" />
|
||||
<span id="evolution" />
|
||||
|
||||
- [Lance format specification](https://lance.org/format): how the format works.
|
||||
- [Multimodal data and blobs](/management/table#multimodal): declare `lancedb.blob()` columns and read blobs on demand.
|
||||
- [Table versioning](/management/table#versioning): read and manage table versions.
|
||||
- [Computed columns](/feature-engineering/add-computed-column): add derived features to your data.
|
||||
- [Architecture](/index#architecture): how LanceDB Embedded and Enterprise build on Lance.
|
||||
|
||||
@@ -126,14 +126,14 @@ tbl.add_columns({"jerk_score": jerk_score}) # nothing computed yet
|
||||
tbl.backfill("jerk_score", concurrency=8) # 610,403 rows in 16.7 s
|
||||
```
|
||||
|
||||
Embeddings work the same way, reading frames straight from the video blobs in the same table:
|
||||
Embeddings work the same way, reading frames straight from the video blobs in the `videos` table:
|
||||
|
||||
```python
|
||||
@udf(data_type=pa.list_(pa.float32(), 768), num_gpus=1,
|
||||
input_columns=["episode_index", "frame_index"])
|
||||
class EmbedFromBlob: # SigLIP2 loads once per worker
|
||||
def __call__(self, episode_index, frame_index):
|
||||
blob = self.videos.take_blobs("video_bytes", indices=[vrow])[0]
|
||||
blob = self.videos.fetch_blob_files("video_bytes", [video_row_id])[0]
|
||||
frame = VideoDecoder(blob).get_frames_at([pos]).data[0]
|
||||
...
|
||||
```
|
||||
|
||||
@@ -63,8 +63,10 @@ uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
If you also need the optional `lance` Python API, install the `pylance` extra and
|
||||
include the Lance preview index:
|
||||
Most applications only need `lancedb`. Install the optional `pylance` extra and
|
||||
include the Lance preview index only when building query engines or connectors,
|
||||
calling format-level `lance` APIs directly, or converting plain-scan lazy or
|
||||
legacy blob columns with `to_pandas()`:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Python (pip) icon="terminal"
|
||||
|
||||
+5
-3
@@ -20,7 +20,7 @@ For those hosts, install the `lancedb-compat` package instead:
|
||||
pip install lancedb-compat
|
||||
```
|
||||
|
||||
Same Python API (`import lancedb` works as usual). The compat wheel is compiled at the `x86-64-v2` baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use `lance.simd_info()` from Python to verify which tier was selected.
|
||||
Same Python API (`import lancedb` works as usual). The compat wheel is compiled at the `x86-64-v2` baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon (if `pylance` is installed, `lance.simd_info()` reports which tier was selected).
|
||||
|
||||
`lancedb` and `lancedb-compat` install to the same `lancedb/` namespace and conflict at install time. Pick one. To switch, `pip uninstall lancedb` first, then `pip install lancedb-compat` (or vice-versa).
|
||||
|
||||
@@ -40,8 +40,10 @@ Stable releases are created about every 2 weeks. For the latest features and bug
|
||||
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
|
||||
```
|
||||
|
||||
If you also need the optional `lance` Python API, install the `pylance` extra
|
||||
and include the Lance preview index:
|
||||
Most applications only need `lancedb`. Install the optional `pylance` extra and
|
||||
include the Lance preview index only when building query engines or connectors,
|
||||
calling format-level `lance` APIs directly, or converting plain-scan lazy or
|
||||
legacy blob columns with `to_pandas()`:
|
||||
|
||||
```bash
|
||||
pip install --pre \
|
||||
|
||||
Reference in New Issue
Block a user