docs: describe refresh and retries the same way on every page

Five pages still said a refresh never revisits a filled row and that a
changed input needs the column dropped and declared again. Since
lancedb#4161 a refresh also recomputes rows whose inputs changed, so the
quickstart, the feature-engineering overview, Guarantees, and the SQL
REFRESH and UPDATE pages now say that.

Retry claims are qualified the same way. A local refresh commits the column
once, and a deployment's job commits finished batches. A retry fills what is
still empty or stale, but can recompute values that were already written (a
refresh whose input record was not saved recomputes next time), so no page
promises that a retry costs only the unfinished rows.
This commit is contained in:
Brendan Clement
2026-10-05 14:12:12 -07:00
parent 69f11c259d
commit 6367f7a0de
6 changed files with 45 additions and 34 deletions
+13 -9
View File
@@ -79,17 +79,21 @@ versions, widen the window before running it, not after.
## Jobs {#jobs}
**A refresh never revisits a filled row.** It gives values to rows that lack
them and leaves the rest alone. Two consequences, both relied on elsewhere:
**A refresh recomputes only what changed.** It gives values to rows that lack
them and recomputes rows whose inputs changed since they were computed; it
leaves the rest alone. Two consequences, both relied on elsewhere:
- **Retrying is safe and cheap.** A failed backfill is resumed, not restarted —
re-running costs only the rows that did not complete.
- **A filled value does not track its input.** If a column derives from `price`
and a row's `price` later changes, the derived value stays as computed.
Recomputing means dropping the column and declaring it again.
- **Retrying is safe.** Re-running a refresh fills what is still empty or stale.
It is not guaranteed to cost only the rows that did not complete: a retry can
recompute values that were already written.
- **A filled value catches up at the next refresh.** If a column derives from
`price` and a row's `price` later changes, the derived value keeps its old
value until the next refresh recomputes it.
**A failure is not a rollback.** Rows filled before a failure keep their values,
because each commits as it completes.
**What a failure keeps depends on where the refresh ran.** `refresh_column` in
your process commits the column once, so a failure keeps none of its values. A
refresh job on a deployment commits finished batches as it goes, so some values
can remain.
**Cancellation is a request.** `cancel()` asks a job to stop; it does not
guarantee it stops before doing more work, and cancelling a finished job does
+16 -14
View File
@@ -31,16 +31,16 @@ separation is the point:
- Filling is a separate call you control, can retry, and -- against a deployment
-- can hand to the server as a [job](/basics/jobs) rather than waiting on.
**Filling never revisits work it has already done.** A refresh gives values to
rows that lack them and leaves the rest alone, so calling it twice is not twice
the work, and appending rows means the next refresh handles only the new ones.
**Filling does not redo current work.** A refresh gives values to rows that
lack them and recomputes rows whose inputs changed since they were computed. It
leaves the rest alone, so calling it twice is not twice the work, and appending
rows means the next refresh handles only the new ones.
The corollary is worth stating plainly: **a filled value is not recomputed when
its inputs change.** If a column derives from `price` and a row's `price` is
later updated, the derived value stays as it was computed. Recomputing means
dropping the column and declaring it again. That is a deliberate trade -- it is
what makes refreshes idempotent and cheap -- but it will surprise you if you
expect a spreadsheet.
The corollary is worth stating plainly: **a filled value catches up with its
inputs only when you refresh.** If a column derives from `price` and a row's
`price` is later updated, the derived value keeps its old value until the next
refresh recomputes it. That is a deliberate trade -- it is what makes refreshes
idempotent and cheap -- but it will surprise you if you expect a spreadsheet.
## Which one to reach for {#which-one}
@@ -85,16 +85,18 @@ That relocation is the main thing to understand about Enterprise feature
engineering. The API is the same; what changes is that the code runs somewhere
else, which is also why a function must declare the environment it needs.
## Failure is resumable {#failures}
## Recovering from a failure {#failures}
A function that raises fails the job. There is no per-row retry or skip policy
in the function definition, so a function calling a flaky external service
should handle that itself, inside the function body.
The alternative is to let the job fail and run the refresh again. Since a
refresh never revisits a filled row, re-running after a partial failure costs
only the rows that did not complete -- **a failed backfill is resumed, not
restarted.**
The alternative is to let the job fail, fix the cause, and run the refresh
again. The new refresh fills what is still empty or stale. How much of the
failed run was kept depends on where it ran (see
[Refresh a column](/feature-engineering/refresh-column#failures)), and a retry
can recompute values that were already written, so do not count on it costing
only the rows that did not complete.
## Where to go next {#next}
@@ -115,8 +115,11 @@ was. A refresh job on a deployment commits finished batches as it goes, so some
values can remain after a failure.
Either way, recover by fixing the cause and refreshing again. The new refresh
fills only rows that are still empty or whose inputs changed, so values that
were kept are not computed twice.
fills rows that are still empty or whose inputs changed, and normally skips
values that were kept. That is not guaranteed: a refresh can recompute values
that were already written -- for example when the record of the inputs they
were computed from was not saved -- so do not count on a retry costing only the
rows that did not complete.
For a column filled by a [function](/management/function), the usual cause is
the worker environment rather than the data. See
+4 -4
View File
@@ -397,10 +397,10 @@ characters.refresh_column("power_score")
# RefreshColumnResult(rows_filled=3, version=4)
```
A refresh only gives values to rows that have none, so appending more characters
and refreshing again fills just the new ones -- it does not recompute the rows
already done. That is what makes a backfill safe to retry and cheap to repeat as
data arrives.
A refresh gives values to rows that have none and recomputes only rows whose
inputs changed, so appending more characters and refreshing again fills just
the new ones -- it does not recompute the rows that are already current. That
is what makes a backfill safe to retry and cheap to repeat as data arrives.
On a deployment the same refresh runs server-side, on a cluster, as a
[job](/basics/jobs) you can walk away from:
+4 -3
View File
@@ -25,9 +25,10 @@ REFRESH COLUMN doubled ON my_table
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
```
A refresh gives values to rows that lack them and **leaves filled rows alone**.
That makes it idempotent, cheap to repeat as data arrives, and resumable after a
failure -- re-running costs only the rows that did not complete.
A refresh gives values to rows that lack them, recomputes rows whose inputs
changed, and **leaves other filled rows alone**. That makes it idempotent, cheap
to repeat as data arrives, and safe to re-run after a failure -- though a re-run
can recompute values the failed run had already written.
`WHERE` scopes the refresh, which is how you fill a slice and check it before
committing to the whole table.
+3 -2
View File
@@ -18,8 +18,9 @@ clause every row is updated.
<Warning>
An update does not recompute [computed columns](/feature-engineering/add-computed-column)
that derive from the column you changed. A filled value keeps what it was
computed from -- see [Guarantees](/basics/guarantees#jobs).
that derive from the column you changed. Those rows keep the value computed from
the old input until the next refresh recomputes them -- see
[Guarantees](/basics/guarantees#jobs).
</Warning>
Commits a new table version, so [versioning](/management/table#versioning) and