Weston Pace 1f1d03f306 feat(python): add backpressure to StreamingDataset post-transform queue (#3897) ...

Rename prefetch_batches → io_queue_depth and introduce
transform_queue_depth as a symmetric pair: both express "number of
batches to buffer per split at this pipeline stage." The old names are
still accepted as keyword arguments but log a deprecation warning
redirecting callers to the new names.

transform_queue_depth caps how many transform-result batches can
accumulate per split in the post-transform queue. Without this limit a
slow consumer (e.g. a GPU training step) causes cooked rows to pile up
unboundedly. The backpressure check in _try_submit_tx counts both
already-cooked rows and rows expected from in-flight transforms; it
skips proactive transform submission when the combined total reaches the
limit. The reactive _ensure_cooked path bypasses the check so the
consumer never stalls.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-08-23 15:28:07 +08:00

..

lancedb

feat(python): add backpressure to StreamingDataset post-transform queue (#3897)

2026-08-23 15:28:07 +08:00

tests

feat(python): add backpressure to StreamingDataset post-transform queue (#3897)

2026-08-23 15:28:07 +08:00

type_tests

fix(python): expose inline types to downstream checkers (#3817)

2026-08-07 17:31:42 +08:00