mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
feat(python): fill failed streaming transform rows
Add `on_transform_error="fill"` to `StreamingDataset`, replacing failed
transform rows at their original positions with a deep-copied fill value or
the result of `repair(exc, row_batch)`.
Fill mode preserves split length, elastic step layout, worker accounting, and
checkpoint progress. Repair failures remain fatal, and `rows_filled` is shared
across workers.
Validated with focused fill and adjacent dataloader tests (10 and 6 passed)
and the complete applicable CI run for
6b51b33e56: four workflows and fifteen checks
all succeeded.
Closes #3916
This commit is contained in:
@@ -778,6 +778,27 @@ so the fetch APIs cannot read their bytes yet.
|
||||
|
||||
::: lancedb.streaming.StreamingDataset
|
||||
|
||||
When a transform fails for some rows, fill mode replaces each failed row in place:
|
||||
|
||||
```python
|
||||
dataset = StreamingDataset(
|
||||
table,
|
||||
num_splits=8,
|
||||
transform=transform,
|
||||
on_transform_error="fill",
|
||||
fill_value={"id": -1, "vector": [0.0, 0.0]},
|
||||
)
|
||||
for row in dataset:
|
||||
train(row)
|
||||
print(dataset.rows_filled)
|
||||
```
|
||||
|
||||
The placeholder must have the same structure as a successful transform output.
|
||||
Pass `repair(exc, row_batch) -> row` instead of `fill_value` to construct a
|
||||
replacement from the failed one-row Arrow batch. Keep the repair deterministic
|
||||
when using elastic training or checkpoints. Fill mode preserves split lengths
|
||||
and sample positions.
|
||||
|
||||
::: lancedb.streaming.StreamingDataLoader
|
||||
|
||||
::: lancedb.permutation.permutation_builder
|
||||
|
||||
Reference in New Issue
Block a user