mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
docs: describe refresh and retries the same way on every page
Five pages still said a refresh never revisits a filled row and that a changed input needs the column dropped and declared again. Since lancedb#4161 a refresh also recomputes rows whose inputs changed, so the quickstart, the feature-engineering overview, Guarantees, and the SQL REFRESH and UPDATE pages now say that. Retry claims are qualified the same way. A local refresh commits the column once, and a deployment's job commits finished batches. A retry fills what is still empty or stale, but can recompute values that were already written (a refresh whose input record was not saved recomputes next time), so no page promises that a retry costs only the unfinished rows.
This commit is contained in:
@@ -79,17 +79,21 @@ versions, widen the window before running it, not after.
|
||||
|
||||
## Jobs {#jobs}
|
||||
|
||||
**A refresh never revisits a filled row.** It gives values to rows that lack
|
||||
them and leaves the rest alone. Two consequences, both relied on elsewhere:
|
||||
**A refresh recomputes only what changed.** It gives values to rows that lack
|
||||
them and recomputes rows whose inputs changed since they were computed; it
|
||||
leaves the rest alone. Two consequences, both relied on elsewhere:
|
||||
|
||||
- **Retrying is safe and cheap.** A failed backfill is resumed, not restarted —
|
||||
re-running costs only the rows that did not complete.
|
||||
- **A filled value does not track its input.** If a column derives from `price`
|
||||
and a row's `price` later changes, the derived value stays as computed.
|
||||
Recomputing means dropping the column and declaring it again.
|
||||
- **Retrying is safe.** Re-running a refresh fills what is still empty or stale.
|
||||
It is not guaranteed to cost only the rows that did not complete: a retry can
|
||||
recompute values that were already written.
|
||||
- **A filled value catches up at the next refresh.** If a column derives from
|
||||
`price` and a row's `price` later changes, the derived value keeps its old
|
||||
value until the next refresh recomputes it.
|
||||
|
||||
**A failure is not a rollback.** Rows filled before a failure keep their values,
|
||||
because each commits as it completes.
|
||||
**What a failure keeps depends on where the refresh ran.** `refresh_column` in
|
||||
your process commits the column once, so a failure keeps none of its values. A
|
||||
refresh job on a deployment commits finished batches as it goes, so some values
|
||||
can remain.
|
||||
|
||||
**Cancellation is a request.** `cancel()` asks a job to stop; it does not
|
||||
guarantee it stops before doing more work, and cancelling a finished job does
|
||||
|
||||
@@ -31,16 +31,16 @@ separation is the point:
|
||||
- Filling is a separate call you control, can retry, and -- against a deployment
|
||||
-- can hand to the server as a [job](/basics/jobs) rather than waiting on.
|
||||
|
||||
**Filling never revisits work it has already done.** A refresh gives values to
|
||||
rows that lack them and leaves the rest alone, so calling it twice is not twice
|
||||
the work, and appending rows means the next refresh handles only the new ones.
|
||||
**Filling does not redo current work.** A refresh gives values to rows that
|
||||
lack them and recomputes rows whose inputs changed since they were computed. It
|
||||
leaves the rest alone, so calling it twice is not twice the work, and appending
|
||||
rows means the next refresh handles only the new ones.
|
||||
|
||||
The corollary is worth stating plainly: **a filled value is not recomputed when
|
||||
its inputs change.** If a column derives from `price` and a row's `price` is
|
||||
later updated, the derived value stays as it was computed. Recomputing means
|
||||
dropping the column and declaring it again. That is a deliberate trade -- it is
|
||||
what makes refreshes idempotent and cheap -- but it will surprise you if you
|
||||
expect a spreadsheet.
|
||||
The corollary is worth stating plainly: **a filled value catches up with its
|
||||
inputs only when you refresh.** If a column derives from `price` and a row's
|
||||
`price` is later updated, the derived value keeps its old value until the next
|
||||
refresh recomputes it. That is a deliberate trade -- it is what makes refreshes
|
||||
idempotent and cheap -- but it will surprise you if you expect a spreadsheet.
|
||||
|
||||
## Which one to reach for {#which-one}
|
||||
|
||||
@@ -85,16 +85,18 @@ That relocation is the main thing to understand about Enterprise feature
|
||||
engineering. The API is the same; what changes is that the code runs somewhere
|
||||
else, which is also why a function must declare the environment it needs.
|
||||
|
||||
## Failure is resumable {#failures}
|
||||
## Recovering from a failure {#failures}
|
||||
|
||||
A function that raises fails the job. There is no per-row retry or skip policy
|
||||
in the function definition, so a function calling a flaky external service
|
||||
should handle that itself, inside the function body.
|
||||
|
||||
The alternative is to let the job fail and run the refresh again. Since a
|
||||
refresh never revisits a filled row, re-running after a partial failure costs
|
||||
only the rows that did not complete -- **a failed backfill is resumed, not
|
||||
restarted.**
|
||||
The alternative is to let the job fail, fix the cause, and run the refresh
|
||||
again. The new refresh fills what is still empty or stale. How much of the
|
||||
failed run was kept depends on where it ran (see
|
||||
[Refresh a column](/feature-engineering/refresh-column#failures)), and a retry
|
||||
can recompute values that were already written, so do not count on it costing
|
||||
only the rows that did not complete.
|
||||
|
||||
## Where to go next {#next}
|
||||
|
||||
|
||||
@@ -115,8 +115,11 @@ was. A refresh job on a deployment commits finished batches as it goes, so some
|
||||
values can remain after a failure.
|
||||
|
||||
Either way, recover by fixing the cause and refreshing again. The new refresh
|
||||
fills only rows that are still empty or whose inputs changed, so values that
|
||||
were kept are not computed twice.
|
||||
fills rows that are still empty or whose inputs changed, and normally skips
|
||||
values that were kept. That is not guaranteed: a refresh can recompute values
|
||||
that were already written -- for example when the record of the inputs they
|
||||
were computed from was not saved -- so do not count on a retry costing only the
|
||||
rows that did not complete.
|
||||
|
||||
For a column filled by a [function](/management/function), the usual cause is
|
||||
the worker environment rather than the data. See
|
||||
|
||||
@@ -397,10 +397,10 @@ characters.refresh_column("power_score")
|
||||
# RefreshColumnResult(rows_filled=3, version=4)
|
||||
```
|
||||
|
||||
A refresh only gives values to rows that have none, so appending more characters
|
||||
and refreshing again fills just the new ones -- it does not recompute the rows
|
||||
already done. That is what makes a backfill safe to retry and cheap to repeat as
|
||||
data arrives.
|
||||
A refresh gives values to rows that have none and recomputes only rows whose
|
||||
inputs changed, so appending more characters and refreshing again fills just
|
||||
the new ones -- it does not recompute the rows that are already current. That
|
||||
is what makes a backfill safe to retry and cheap to repeat as data arrives.
|
||||
|
||||
On a deployment the same refresh runs server-side, on a cluster, as a
|
||||
[job](/basics/jobs) you can walk away from:
|
||||
|
||||
@@ -25,9 +25,10 @@ REFRESH COLUMN doubled ON my_table
|
||||
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
||||
```
|
||||
|
||||
A refresh gives values to rows that lack them and **leaves filled rows alone**.
|
||||
That makes it idempotent, cheap to repeat as data arrives, and resumable after a
|
||||
failure -- re-running costs only the rows that did not complete.
|
||||
A refresh gives values to rows that lack them, recomputes rows whose inputs
|
||||
changed, and **leaves other filled rows alone**. That makes it idempotent, cheap
|
||||
to repeat as data arrives, and safe to re-run after a failure -- though a re-run
|
||||
can recompute values the failed run had already written.
|
||||
|
||||
`WHERE` scopes the refresh, which is how you fill a slice and check it before
|
||||
committing to the whole table.
|
||||
|
||||
@@ -18,8 +18,9 @@ clause every row is updated.
|
||||
|
||||
<Warning>
|
||||
An update does not recompute [computed columns](/feature-engineering/add-computed-column)
|
||||
that derive from the column you changed. A filled value keeps what it was
|
||||
computed from -- see [Guarantees](/basics/guarantees#jobs).
|
||||
that derive from the column you changed. Those rows keep the value computed from
|
||||
the old input until the next refresh recomputes them -- see
|
||||
[Guarantees](/basics/guarantees#jobs).
|
||||
</Warning>
|
||||
|
||||
Commits a new table version, so [versioning](/management/table#versioning) and
|
||||
|
||||
Reference in New Issue
Block a user