lancedb/docs/web/integrations/lerobotdataset.mdx

---
title: "LeRobotDataset"
sidebarTitle: "LeRobotDataset"
description: "Train LeRobot policies straight from object storage, and curate, search and visualize robotics datasets as Lance tables."
keywords: ["lerobot", "robotics", "droid", "training", "dataloader"]
---

import {
  PyFrameworksLerobotFilterFrames,
  PyFrameworksLerobotLoad,
  PyFrameworksLerobotOpenLanceTables,
  PyFrameworksLerobotTrainReady,
} from '/snippets/integrations.mdx';

[LeRobot](https://huggingface.co/docs/lerobot/index) is Hugging Face's open-source robotics stack. Since [lerobot#4363](https://github.com/huggingface/lerobot/pull/4363) it reads Lance datasets natively: same `LeRobotDataset` class, same items, same training pipeline. The dataset's `"storage_format": "lance"` picks the reader. No subclasses, no wrapper library.

What you get:

- **Train from object storage while keeping your GPUs fed.** Lance reads random frames straight from S3, GCS, the HF Hub or HF Storage Buckets, fetching only the bytes each batch needs, so every batch is a global shuffle: any frame, from anywhere in the dataset. In the table below (samples/s, 8 workers, batch 64), Lance and upstream local both draw every batch from the whole dataset. Upstream streaming does a reservoir shuffle, reading the dataset in order and drawing each batch from a pool of recently seen frames, so those two columns are not the same job:

| dataset | window shuffle, Lance from object store | window shuffle, upstream streaming from the Hub | Lance (object store) vs upstream streaming | RAM, Lance | RAM, streaming | global shuffle, Lance from object store | global shuffle, upstream from local NVMe | Lance (object store) vs upstream (local NVMe) |
|---|---|---|---|---|---|---|---|---|
| DROID | 904 | 478 | 1.9x | 8 GB | 122 GB | 257 | 224 | 1.1x |
| roboturk | 778 | 152 | 5.1x | 6 GB | 9 GB | 1,074 | 983 | 1.1x |
| aloha_mobile_cabinet | 772 | 51 | 15.2x | 9 GB | 30 GB | 693 | 573 | 1.2x |
| koch_pick_place_5_lego | 701 | 54 | 13.0x | 8 GB | 15 GB | 876 | 526 | 1.7x |
| pusht | 5,340 | 648 | 8.2x | 3 GB | 7 GB | 5,508 | 4,224 | 1.3x |
| aloha_static_cups_open | 597 | 45 | 13.2x | 10 GB | 38 GB | 575 | 483 | 1.2x |

The shuffle window is 0.05% of the dataset, at least 5,000 frames: 5,000 on the small sets, 13,815 on DROID. Streaming DROID at that window in upstream default needs 122 GB, so those two runs used a 256 GB machine with the same decode speed. The DROID pair is noisy: Lance 684 and 1,124, streaming 303 and 653.

Same batches: Lance from an object store is 1.9x faster than upstream streaming on DROID and 5x to 15x on the other datasets, with 8 GB of RAM against 122 GB on DROID. Global shuffle: Lance with nothing downloaded is 1.1x to 1.7x faster than upstream reading local NVMe after the download. berkeley_rpt, toto and soarm101 are left out because their metadata lists frames the videos do not have; `lerobot-lance-doctor` reports that before you train.

- **Faster iteration, end to end.** 10,000 steps of SmolVLA on DROID, 8xH100, lerobot's default `num_workers=4`, same seed:

| | Lance, S3, nothing downloaded | upstream, local NVMe after a 384 GB download |
|---|---|---|
| wall clock | **1 h 27 m** | 2 h 00 m |
| steady rate | 495 samples/s | 361 samples/s |
| of each step spent waiting on data | 1.7% | 37.4% |
| final loss | 0.2380 | 0.2380 |

Same model, same loss, 33 minutes sooner, and no 384 GB download first. The difference is time the GPUs spend waiting on data, so it grows with the run: at 100k steps it is over five hours, and every re-run and sweep point pays it again.

- **The training table is also the index.** Parquet has no secondary indexes, so finding frames instead of scanning them usually means another system per question: a vector database for embeddings, a search service for instructions, a feature store for scores. With Lance those are columns and indexes on the same table the DataLoader reads. One query spans all of them, at a version the trainer can reopen.

## Install

```bash
pip install "lerobot[lancedb]"   # the reader, inside lerobot itself
pip install lerobot-lancedb      # the converter
```

## Convert a dataset

Any LeRobot v3.0 dataset, from the Hub or a local directory:

```bash
lerobot-lance-convert --repo-id lerobot/pusht --out ./pusht-lance
```

**Migration costs about as much as a file copy.** Videos are stored as blobs without re-encoding, so items are bit-identical to the source: 392,578 frames convert in 10 seconds, DROID's 27.6M in 34 minutes. Nothing about your recording or training code changes. The output keeps `meta/` as is and adds three Lance tables (`frames`, `videos`, `meta`). `lerobot-lance-doctor` audits the result and often finds problems in the source: DROID carries 44% orphaned rows, and three of eight public datasets we converted have frame-count defects that only random access exposes.

Host the result anywhere: an HF dataset repo (`hf upload`, tag it `v3.0`), an HF Storage Bucket (`hf buckets sync`), or any object store (`aws s3 sync`).

## Load and train

<CodeBlock filename="Python" language="Python" icon="python">
  {PyFrameworksLerobotLoad}
</CodeBlock>

An explicit `root` works for any object store: `LeRobotDataset("lerobot/pusht", root="s3://my-bucket/pusht-lance")`. Remote roots download only `meta/`. Data is fetched per batch. Temporal windows, `DataLoader` and the rest of the training stack work unchanged:

<CodeBlock filename="Python" language="Python" icon="python">
  {PyFrameworksLerobotTrainReady}
</CodeBlock>

Or from the CLI. Held-out evaluation on a remote dataset needs random access, so it only works on the Lance path:

```bash
lerobot-train \
  --dataset.repo_id=my-org/my-bucket/pusht-lance \
  --dataset.repo_type=bucket \
  --dataset.eval_split=0.02 \
  --policy.type=act
```

## Visualize TB-scale datasets in Foxglove

LeRobot's dataset viewer can serve an episode to [Foxglove](https://foxglove.dev) over its WebSocket protocol ([lerobot#4542](https://github.com/huggingface/lerobot/pull/4542)), and with a Lance dataset it works wherever the data lives:

```bash
lerobot-dataset-viz --repo-id lance-format/pusht-lance --episode-index 3 --display-mode foxglove
```

Open the Foxglove app, connect to `ws://localhost:8765`, and scrub. Cameras, joint state and actions show up as topics you can lay out however you like:

![Foxglove showing episode 7 of DROID served from S3: three camera panels and the joint state, scrubbed to 19.4 s](/static/assets/images/integrations/lerobot-foxglove-droid.png)

Frames are fetched as you scrub, so looking at one episode of the 373 GB DROID from S3 moves a few MB, not the dataset. No MCAP export, no download, and nothing new to install on the dataset side: this is the same reader the trainer uses.

## Query, curate, enrich

The frames table is a LanceDB table like any other:

<CodeBlock filename="Python" language="Python" icon="python">
  {PyFrameworksLerobotOpenLanceTables}
</CodeBlock>

Filter with SQL over any feature, using the indexes built at conversion:

<CodeBlock filename="Python" language="Python" icon="python">
  {PyFrameworksLerobotFilterFrames}
</CodeBlock>

From here it is standard LanceDB. The examples below are from our DROID walkthrough, all on the table the trainer reads.

**Feature engineering costs nothing until you use it.** A new column is a [declaration](/feature-engineering/add-computed-column): `add_columns` registers the expression or [function](/management/function) and computes nothing, and a [refresh](/feature-engineering/refresh-column) materializes it in parallel when you want it. No copy of the dataset, and the videos are never touched:

```python
@udf(data_type=pa.float32(), input_columns=["action_joint_velocity"])
def jerk_score(v):                      # per-frame motion roughness
    ...

tbl.add_columns({"jerk_score": jerk_score})   # nothing computed yet
tbl.backfill("jerk_score", concurrency=8)     # 610,403 rows in 16.7 s
```

Embeddings work the same way, reading frames straight from the video blobs in the same table:

```python
@udf(data_type=pa.list_(pa.float32(), 768), num_gpus=1,
     input_columns=["episode_index", "frame_index"])
class EmbedFromBlob:                    # SigLIP2 loads once per worker
    def __call__(self, episode_index, frame_index):
        blob = self.videos.take_blobs("video_bytes", indices=[vrow])[0]
        frame = VideoDecoder(blob).get_frames_at([pos]).data[0]
        ...
```

**Search** is an index on those columns: a vector index over the embeddings built in 29.7 s across 27.6M rows, a full-text index over the instructions in 7.0 s, queries in 10 to 54 ms.

**Curation** is one query across all of it. Semantic search with a predicate on the derived score, no join, no second system:

```python
tbl.search(vec("a gripper closing on an object"), vector_column_name="emb_siglip2") \
   .where("jerk_score > 1.2875")        # the roughest 1% of frames
   .limit(4)
```

**Mining** is the same query pointed at the rare cases. `tbl.search().where("jerk_score > 1.2875 AND success = false")` pulls the rough, failed frames out of 27.6M rows in milliseconds, ready to become a fine-tuning set or a labeling queue.

**EDA** is a scan of the tabular columns: a pass over all 27.6M rows reads 2.52 GB in 190 s and decodes no video.

Then drop what you found from training with `episodes=[...]` instead of rewriting files. The reader ignores columns it does not know, and the table is versioned, so curation and training share one copy of the data at a version you can reopen.

<Info>
Lance datasets are read-only in LeRobot: record and edit in the default format, then convert.
</Info>

Public examples to try everything above: [`lance-format/pusht-lance`](https://huggingface.co/datasets/lance-format/pusht-lance) and friends under the [`lance-format`](https://huggingface.co/lance-format) org.