diff --git a/docs/web/basics/guarantees.mdx b/docs/web/basics/guarantees.mdx index 59a89c32d..47c7c8d7c 100644 --- a/docs/web/basics/guarantees.mdx +++ b/docs/web/basics/guarantees.mdx @@ -79,17 +79,21 @@ versions, widen the window before running it, not after. ## Jobs {#jobs} -**A refresh never revisits a filled row.** It gives values to rows that lack -them and leaves the rest alone. Two consequences, both relied on elsewhere: +**A refresh recomputes only what changed.** It gives values to rows that lack +them and recomputes rows whose inputs changed since they were computed; it +leaves the rest alone. Two consequences, both relied on elsewhere: -- **Retrying is safe and cheap.** A failed backfill is resumed, not restarted — - re-running costs only the rows that did not complete. -- **A filled value does not track its input.** If a column derives from `price` - and a row's `price` later changes, the derived value stays as computed. - Recomputing means dropping the column and declaring it again. +- **Retrying is safe.** Re-running a refresh fills what is still empty or stale. + It is not guaranteed to cost only the rows that did not complete: a retry can + recompute values that were already written. +- **A filled value catches up at the next refresh.** If a column derives from + `price` and a row's `price` later changes, the derived value keeps its old + value until the next refresh recomputes it. -**A failure is not a rollback.** Rows filled before a failure keep their values, -because each commits as it completes. +**What a failure keeps depends on where the refresh ran.** `refresh_column` in +your process commits the column once, so a failure keeps none of its values. A +refresh job on a deployment commits finished batches as it goes, so some values +can remain. **Cancellation is a request.** `cancel()` asks a job to stop; it does not guarantee it stops before doing more work, and cancelling a finished job does diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index 253f9734f..53fea63c3 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -31,16 +31,16 @@ separation is the point: - Filling is a separate call you control, can retry, and -- against a deployment -- can hand to the server as a [job](/basics/jobs) rather than waiting on. -**Filling never revisits work it has already done.** A refresh gives values to -rows that lack them and leaves the rest alone, so calling it twice is not twice -the work, and appending rows means the next refresh handles only the new ones. +**Filling does not redo current work.** A refresh gives values to rows that +lack them and recomputes rows whose inputs changed since they were computed. It +leaves the rest alone, so calling it twice is not twice the work, and appending +rows means the next refresh handles only the new ones. -The corollary is worth stating plainly: **a filled value is not recomputed when -its inputs change.** If a column derives from `price` and a row's `price` is -later updated, the derived value stays as it was computed. Recomputing means -dropping the column and declaring it again. That is a deliberate trade -- it is -what makes refreshes idempotent and cheap -- but it will surprise you if you -expect a spreadsheet. +The corollary is worth stating plainly: **a filled value catches up with its +inputs only when you refresh.** If a column derives from `price` and a row's +`price` is later updated, the derived value keeps its old value until the next +refresh recomputes it. That is a deliberate trade -- it is what makes refreshes +idempotent and cheap -- but it will surprise you if you expect a spreadsheet. ## Which one to reach for {#which-one} @@ -85,16 +85,18 @@ That relocation is the main thing to understand about Enterprise feature engineering. The API is the same; what changes is that the code runs somewhere else, which is also why a function must declare the environment it needs. -## Failure is resumable {#failures} +## Recovering from a failure {#failures} A function that raises fails the job. There is no per-row retry or skip policy in the function definition, so a function calling a flaky external service should handle that itself, inside the function body. -The alternative is to let the job fail and run the refresh again. Since a -refresh never revisits a filled row, re-running after a partial failure costs -only the rows that did not complete -- **a failed backfill is resumed, not -restarted.** +The alternative is to let the job fail, fix the cause, and run the refresh +again. The new refresh fills what is still empty or stale. How much of the +failed run was kept depends on where it ran (see +[Refresh a column](/feature-engineering/refresh-column#failures)), and a retry +can recompute values that were already written, so do not count on it costing +only the rows that did not complete. ## Where to go next {#next} diff --git a/docs/web/feature-engineering/refresh-column.mdx b/docs/web/feature-engineering/refresh-column.mdx index 237da3491..d0f9f06a3 100644 --- a/docs/web/feature-engineering/refresh-column.mdx +++ b/docs/web/feature-engineering/refresh-column.mdx @@ -115,8 +115,11 @@ was. A refresh job on a deployment commits finished batches as it goes, so some values can remain after a failure. Either way, recover by fixing the cause and refreshing again. The new refresh -fills only rows that are still empty or whose inputs changed, so values that -were kept are not computed twice. +fills rows that are still empty or whose inputs changed, and normally skips +values that were kept. That is not guaranteed: a refresh can recompute values +that were already written -- for example when the record of the inputs they +were computed from was not saved -- so do not count on a retry costing only the +rows that did not complete. For a column filled by a [function](/management/function), the usual cause is the worker environment rather than the data. See diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index 50dc72945..79a94754b 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -397,10 +397,10 @@ characters.refresh_column("power_score") # RefreshColumnResult(rows_filled=3, version=4) ``` -A refresh only gives values to rows that have none, so appending more characters -and refreshing again fills just the new ones -- it does not recompute the rows -already done. That is what makes a backfill safe to retry and cheap to repeat as -data arrives. +A refresh gives values to rows that have none and recomputes only rows whose +inputs changed, so appending more characters and refreshing again fills just +the new ones -- it does not recompute the rows that are already current. That +is what makes a backfill safe to retry and cheap to repeat as data arrives. On a deployment the same refresh runs server-side, on a cluster, as a [job](/basics/jobs) you can walk away from: diff --git a/docs/web/sql/dml/refresh.mdx b/docs/web/sql/dml/refresh.mdx index 5d0101097..3fece1f70 100644 --- a/docs/web/sql/dml/refresh.mdx +++ b/docs/web/sql/dml/refresh.mdx @@ -25,9 +25,10 @@ REFRESH COLUMN doubled ON my_table REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01' ``` -A refresh gives values to rows that lack them and **leaves filled rows alone**. -That makes it idempotent, cheap to repeat as data arrives, and resumable after a -failure -- re-running costs only the rows that did not complete. +A refresh gives values to rows that lack them, recomputes rows whose inputs +changed, and **leaves other filled rows alone**. That makes it idempotent, cheap +to repeat as data arrives, and safe to re-run after a failure -- though a re-run +can recompute values the failed run had already written. `WHERE` scopes the refresh, which is how you fill a slice and check it before committing to the whole table. diff --git a/docs/web/sql/dml/update.mdx b/docs/web/sql/dml/update.mdx index cef35ef4e..042ef91b9 100644 --- a/docs/web/sql/dml/update.mdx +++ b/docs/web/sql/dml/update.mdx @@ -18,8 +18,9 @@ clause every row is updated. An update does not recompute [computed columns](/feature-engineering/add-computed-column) -that derive from the column you changed. A filled value keeps what it was -computed from -- see [Guarantees](/basics/guarantees#jobs). +that derive from the column you changed. Those rows keep the value computed from +the old input until the next refresh recomputes them -- see +[Guarantees](/basics/guarantees#jobs). Commits a new table version, so [versioning](/management/table#versioning) and