mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-31 07:46:51 +00:00
docs(python): document connection_factory on StreamingDataset
Closes #3693. `columns` is documented on `StreamingDataset` now, but `connection_factory` still wasn't. This adds it to the parameter list, in signature order. The text follows `__getstate__` / `__setstate__`: with a factory, pickling the dataset replaces the table's reopen state with its name, and each DataLoader worker calls the factory to reopen the table. The factory is pickled along with the rest of the dataset, so it has to be picklable, and credentials stay out of the pickle only when the factory loads them in the worker instead of capturing them (the same guidance as `Permutation.with_connection_factory`). Without a factory, the table's own reopen state is pickled. This picks up #3697, which documented the same parameter and was closed by its author without a merge.
This commit is contained in:
@@ -395,6 +395,16 @@ class StreamingDataset(IterableDataset):
|
||||
capping peak memory when the consumer (e.g. a GPU training step) is
|
||||
slower than the transform stage. Must be greater than zero.
|
||||
``None`` (the default) imposes no limit.
|
||||
connection_factory:
|
||||
Optional picklable function that takes the table name and returns an
|
||||
open LanceDB table. When set, pickling the dataset (as DataLoader
|
||||
workers do under the ``spawn`` and ``forkserver`` start methods)
|
||||
replaces the table's reopen state with its name, and each worker calls
|
||||
the factory with that name to reopen the table. The factory itself is
|
||||
pickled too, so credentials stay out of the pickle only if the factory
|
||||
loads them in the worker, for example from the environment, instead of
|
||||
capturing them. ``None`` (the default) pickles the table's own reopen
|
||||
state instead.
|
||||
worker_info_override:
|
||||
If set, used in place of ``torch.utils.data.get_worker_info()`` to
|
||||
determine the DataLoader worker assignment. Intended for unit tests
|
||||
|
||||
Reference in New Issue
Block a user