mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Yang Cen
fa4b14a9d6
The refresh guide and SQL reference do not warn that a `WHERE` refresh recomputes filled rows, and the guide describes resources as per-worker allocations. Clarify the aggregate Function budget and expression-only `num_workers`, show the output-null predicate with its stale-value and successful-null limitations in both references, and document refresh-level `error_policy='skip'`. Link both guides to the current SQL options route. Validation: Vale 3.24.0 found no issues on changed lines; `git diff --check` and the new link targets passed. Mint 4.2.888 reports the same four existing `RsExampleBody`/`RsExampleTail` import warnings in `reranking/index.mdx` and `search/vector-search.mdx` on both pristine base `5e38120b` and this branch. No additional warnings. Behavior was checked against the owning Sophon implementation; no runtime code changed or runtime tests were run. Companion to https://github.com/lancedb/sophon/pull/9806 for ENT-3279. Page visibility and saved documentation editions are unchanged. DOC-90 retains publication and matching-shipped-release verification.
112 lines
4.7 KiB
Plaintext
112 lines
4.7 KiB
Plaintext
---
|
|
title: "REFRESH COLUMN"
|
|
sidebarTitle: "REFRESH COLUMN"
|
|
description: "Fill computed outputs and select rows to recompute."
|
|
icon: arrows-rotate
|
|
keywords: ["refresh column", "backfill", "computed column"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A LanceDB extension with no DataFusion equivalent. It is what makes
|
|
[feature engineering](/feature-engineering) reachable from SQL: a column
|
|
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN <column> ON [TABLE] <table>
|
|
[WHERE <predicate>]
|
|
[WITH (<option> = <value> [, ...])]
|
|
```
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN doubled ON my_table
|
|
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
|
```
|
|
</CodeGroup>
|
|
|
|
Without `WHERE`, a refresh fills unassigned rows and recomputes values whose
|
|
inputs changed, leaving other current values alone. A retry after a failure
|
|
can still repeat work that was not committed.
|
|
|
|
With `WHERE`, a refresh **recomputes every matching live row**, including rows
|
|
that already have current values. Repeating a predicate can repeat the Function
|
|
calls and their cost. To select only null outputs within a slice:
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN embedding ON documents
|
|
WHERE created_at > '2026-01-01' AND embedding IS NULL
|
|
```
|
|
</CodeGroup>
|
|
|
|
This skips non-null outputs even if their inputs changed. It also selects
|
|
outputs that were successfully computed as `NULL`. Use an unfiltered refresh
|
|
to let LanceDB track which outputs need updating.
|
|
|
|
## Options {#options}
|
|
|
|
| Option | Meaning |
|
|
| --- | --- |
|
|
| `num_workers` | fixed worker count for SQL-expression columns; rejected for registered Function columns |
|
|
| `cpu_millis` | aggregate CPU ceiling across the refresh's Function workers, in thousandths of a CPU |
|
|
| `memory_bytes` | aggregate memory ceiling across the refresh's Function workers, in bytes |
|
|
| `gpu_count` | aggregate GPU budget across the refresh's workers. Function columns only |
|
|
| `gpu_class` | GPU class name for the refresh's workers. Function columns only |
|
|
| `batch_size` | maximum rows passed to one Function call; actual calls can be smaller. Function columns only |
|
|
| `max_executor_processes_per_worker` | cap on executor processes per worker. Function columns only |
|
|
| `row_timeout_seconds` | deadline for one Function call. Function columns only |
|
|
| `error_policy` | `'fail'` (default) stops on a row error; `'skip'` writes null outputs and records row errors, subject to failure limits. Function columns only |
|
|
| `max_failed_fraction` | fail the job once more than this share of rows has failed. Function columns only |
|
|
| `max_failed_rows` | fail the job once more than this many rows has failed. Function columns only |
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN embedding ON documents
|
|
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
|
|
```
|
|
</CodeGroup>
|
|
|
|
The example allows up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the
|
|
Function workers together. It does not reserve those resources per worker or
|
|
guarantee a worker count. When setting CPU or memory, provide all three budget
|
|
fields. A complete CPU-only budget uses `gpu_count = 0`.
|
|
|
|
Set `error_policy` on the refresh, not on the Function definition. With `skip`,
|
|
inspect failures with `SHOW ERRORS` or `table.function_errors()` and retry the
|
|
affected rows after fixing the cause. Initialization, infrastructure, and
|
|
publication failures can still fail the job.
|
|
|
|
<Note>
|
|
`num_workers` is refused for a column filled by a registered
|
|
[function](/management/function), which sizes its own workers from the CPU,
|
|
memory and GPU options instead. `error_policy` (and the Function-only options
|
|
above it in the table) is the reverse: it's refused for a plain SQL-expression
|
|
column, which has no per-row failures to skip -- it's accepted only for a
|
|
Function column.
|
|
</Note>
|
|
|
|
## Watching a refresh {#watching}
|
|
|
|
A refresh submits work that outlives the call. Track it with
|
|
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
|
[`SHOW ERRORS`](/sql/ddl/job#errors). For per-split progress, worker fleet
|
|
detail, and a `DESCRIBE JOB` example, see [Watching a refresh](/sql/dml/refresh#watching).
|
|
|
|
## See also {#see-also}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
The same operation from the SDKs, and what a refresh guarantees.
|
|
</Card>
|
|
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
|
|
Declaring the columns this fills.
|
|
</Card>
|
|
<Card title="REFRESH MATERIALIZED VIEW" icon="arrows-rotate" href="/sql/dml/refresh-materialized-view">
|
|
Bringing a view's stored result up to date, instead of a column.
|
|
</Card>
|
|
</CardGroup>
|