buduoqiu 98a52267a2 feat(python): configure streaming transform parallelism (#3699) ...

## Summary

- add a keyword-only `transform_parallelism` option to
`StreamingDataset`
- preserve CPU auto-detection by default and fall back to one worker
when unavailable
- apply the configured limit to both the transform executor and
concurrency semaphore
- document and test explicit, default, fallback, and invalid values

## Testing

- `uv run --extra tests --with torch pytest
python/tests/test_elastic_dataloader.py -q` (`136 passed`)
- `uvx ruff check python/lancedb/streaming.py
python/tests/test_elastic_dataloader.py`
- `uvx ruff format --check python/lancedb/streaming.py
python/tests/test_elastic_dataloader.py`
- `git diff --check origin/main...HEAD`

Closes #3695

Co-authored-by: buduoqiu <yaodong-shen@users.noreply.github.com>

2026-07-28 15:38:34 -07:00

..

lancedb

feat(python): configure streaming transform parallelism (#3699)

2026-07-28 15:38:34 -07:00

tests

feat(python): configure streaming transform parallelism (#3699)

2026-07-28 15:38:34 -07:00