mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-22 19:52:32 +00:00
Lei XuandClaude Sonnet 5.5
bedb6ab3b1
## Summary - Rework `docs/web/training/torch.mdx` for people who already know the PyTorch `DataLoader`, built around `lancedb.streaming.StreamingDataset`. - The quickstart trains a state-to-action policy on the real `lance-format/lerobot-xvla-soft-fold` dataset loaded via `hf://`, using a zero-copy Arrow-to-tensor transform. - Add a short table mapping PyTorch idioms (`IterableDataset`, `DistributedSampler`, `collate_fn`, `random_split`, `StatefulDataLoader`) to their LanceDB equivalents. - Add an Optimizations section: Transform (including an `ExtractFrames` example using blob v2 and torchcodec), Distributed Training, and the existing Output Formats content moved underneath. ## Notes - The quickstart depends on #4602 (permutation building over tables with schema metadata), which is merged. - `ExtractFrames` takes a URI and opens the table lazily because a local `LanceTable` cannot yet be pickled into DataLoader workers. - The Permutation-based "Selecting columns" and "Using multiple DataLoader workers" sections further down are unchanged from main. ## Test plan - [x] Ran the quickstart, transform, `ExtractFrames` (main process and forkserver workers) and distributed snippets locally. - [x] Rendered the page with `mint dev` and checked for console errors. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>