feat(python): fill failed streaming transform rows

Add `on_transform_error="fill"` to `StreamingDataset`, replacing failed
transform rows at their original positions with a deep-copied fill value or
the result of `repair(exc, row_batch)`.

Fill mode preserves split length, elastic step layout, worker accounting, and
checkpoint progress. Repair failures remain fatal, and `rows_filled` is shared
across workers.

Validated with focused fill and adjacent dataloader tests (10 and 6 passed)
and the complete applicable CI run for
6b51b33e56: four workflows and fifteen checks
all succeeded.

Closes #3916
This commit is contained in:
importcpp
2026-10-31 12:32:16 +08:00
committed by GitHub
parent 5f7ec4e0a2
commit e56150568c
3 changed files with 267 additions and 18 deletions
+21
View File
@@ -778,6 +778,27 @@ so the fetch APIs cannot read their bytes yet.
::: lancedb.streaming.StreamingDataset
When a transform fails for some rows, fill mode replaces each failed row in place:
```python
dataset = StreamingDataset(
table,
num_splits=8,
transform=transform,
on_transform_error="fill",
fill_value={"id": -1, "vector": [0.0, 0.0]},
)
for row in dataset:
train(row)
print(dataset.rows_filled)
```
The placeholder must have the same structure as a successful transform output.
Pass `repair(exc, row_batch) -> row` instead of `fill_value` to construct a
replacement from the failed one-row Arrow batch. Keep the repair deterministic
when using elastic training or checkpoints. Fill mode preserves split lengths
and sample positions.
::: lancedb.streaming.StreamingDataLoader
::: lancedb.permutation.permutation_builder