lancedb-gatefixer[bot] 105fd73bc6 fix(python): commit streaming worker checkpoints on consumption (#4023) ...

## Summary

- add `StreamingDataLoader`, which transports worker snapshots with
prefetched batches and commits them to the parent dataset only when the
trainer receives each batch
- preserve exact non-uniform per-split progress and resume lagging
splits without replaying already-consumed rows
- reject stale parent checkpoints after a standard multi-process
`DataLoader` has started, with guidance to use the consumer-aware loader
- document the new public loader and merge non-uniform state across
ranks

## Root cause

PyTorch runs `StreamingDataset.__iter__` in private worker-process
copies, while callers invoke `state_dict()` on the parent dataset.
Sharing producer counters would still be incorrect because DataLoader
prefetch can advance workers beyond batches returned to the trainer.

## Validation

- `uv run --extra tests pytest python/tests/test_elastic_dataloader.py
-q` (154 passed)
- focused non-uniform merge regression (1 passed)
- `uv run --project python --extra tests --extra dev ruff format .`
- `uv run --project python --extra tests --extra dev ruff check .`
- `cd docs && PYTHONPATH=. ../python/.venv/bin/mkdocs build`

Fixes #3967

<!-- lance-gatekeeper-fix:v1 agent=572be272619660b97e87fd5c85188341
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>

2026-08-25 04:22:22 +08:00

..

lancedb

fix(python): commit streaming worker checkpoints on consumption (#4023)

2026-08-25 04:22:22 +08:00

tests

fix(python): commit streaming worker checkpoints on consumption (#4023)

2026-08-25 04:22:22 +08:00

type_tests

fix(python): expose inline types to downstream checkers (#3817)

2026-08-07 17:31:42 +08:00