From 65954bf9ccee3e1bedf7e1edbbe1f261c7de270f Mon Sep 17 00:00:00 2001 From: Bilal Atique <74999164+BilalAtique@users.noreply.github.com> Date: Thu, 29 Oct 2026 06:52:15 +0500 Subject: [PATCH] docs(python): document connection_factory on StreamingDataset Closes #3693. `columns` is documented on `StreamingDataset` now, but `connection_factory` still wasn't. This adds it to the parameter list, in signature order. The text follows `__getstate__` / `__setstate__`: with a factory, pickling the dataset replaces the table's reopen state with its name, and each DataLoader worker calls the factory to reopen the table. The factory is pickled along with the rest of the dataset, so it has to be picklable, and credentials stay out of the pickle only when the factory loads them in the worker instead of capturing them (the same guidance as `Permutation.with_connection_factory`). Without a factory, the table's own reopen state is pickled. This picks up #3697, which documented the same parameter and was closed by its author without a merge. --- python/python/lancedb/streaming.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/python/python/lancedb/streaming.py b/python/python/lancedb/streaming.py index 4a4ffd1f5..db71840c6 100644 --- a/python/python/lancedb/streaming.py +++ b/python/python/lancedb/streaming.py @@ -395,6 +395,16 @@ class StreamingDataset(IterableDataset): capping peak memory when the consumer (e.g. a GPU training step) is slower than the transform stage. Must be greater than zero. ``None`` (the default) imposes no limit. + connection_factory: + Optional picklable function that takes the table name and returns an + open LanceDB table. When set, pickling the dataset (as DataLoader + workers do under the ``spawn`` and ``forkserver`` start methods) + replaces the table's reopen state with its name, and each worker calls + the factory with that name to reopen the table. The factory itself is + pickled too, so credentials stay out of the pickle only if the factory + loads them in the worker, for example from the environment, instead of + capturing them. ``None`` (the default) pickles the table's own reopen + state instead. worker_info_override: If set, used in place of ``torch.utils.data.get_worker_info()`` to determine the DataLoader worker assignment. Intended for unit tests