mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
Brendan Clement
eedde82253
Sophon main now refreshes an eligible materialized view incrementally, appending what an append-only source added since the last refresh, a day after the change that made every refresh a full rebuild. So the materialized view page and the REFRESH reference no longer say recent versions rebuild in full: they say recent versions run views as SQL jobs, need no stable row ids and take no refresh options, and that a full or incremental refresh depends on the version and the view's query. COPY is described with the same "recent versions" wording.
101 lines
3.4 KiB
Plaintext
101 lines
3.4 KiB
Plaintext
---
|
|
title: "REFRESH"
|
|
sidebarTitle: "REFRESH"
|
|
description: "Filling computed columns and bringing materialized views up to date."
|
|
icon: arrows-rotate
|
|
keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
|
|
|
LanceDB extensions with no DataFusion equivalent. They are what makes
|
|
[feature engineering](/feature-engineering) reachable from SQL: a column
|
|
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
|
|
|
|
## REFRESH COLUMN {#refresh-column}
|
|
|
|
```sql
|
|
REFRESH COLUMN <column> ON [TABLE] <table>
|
|
[WHERE <predicate>]
|
|
[WITH (<option> = <value> [, ...])]
|
|
```
|
|
|
|
```sql
|
|
REFRESH COLUMN doubled ON my_table
|
|
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
|
```
|
|
|
|
A refresh gives values to rows that lack them, recomputes rows whose inputs
|
|
changed, and **leaves other filled rows alone**. That makes it idempotent, cheap
|
|
to repeat as data arrives, and safe to re-run after a failure -- though a re-run
|
|
can recompute values the failed run had already written.
|
|
|
|
`WHERE` scopes the refresh, which is how you fill a slice and check it before
|
|
committing to the whole table.
|
|
|
|
### Options {#refresh-column-options}
|
|
|
|
| Option | Meaning |
|
|
| --- | --- |
|
|
| `num_workers` | how many workers to use |
|
|
| `cpu_millis` | CPU budget per worker |
|
|
| `memory_bytes` | memory budget per worker |
|
|
| `gpu_count` | GPUs per worker |
|
|
| `batch_size` | rows a worker takes at a time |
|
|
| `error_policy` | `'skip'` continues past failing rows instead of failing the job |
|
|
|
|
```sql
|
|
REFRESH COLUMN embedding ON documents
|
|
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
|
|
```
|
|
|
|
`num_workers` sets a count; the budget options size each worker. They are
|
|
alternatives rather than a pair.
|
|
|
|
<Note>
|
|
`error_policy` and `num_workers` are refused for a column filled by a registered
|
|
[function](/management/function) with its own resource budget -- use the CPU,
|
|
memory and GPU options there.
|
|
</Note>
|
|
|
|
## REFRESH MATERIALIZED VIEW {#refresh-materialized-view}
|
|
|
|
```sql
|
|
REFRESH MATERIALIZED VIEW <name> [CASCADE] [WITH (<option> = <value>)]
|
|
```
|
|
|
|
```sql
|
|
REFRESH MATERIALIZED VIEW chunks
|
|
REFRESH MATERIALIZED VIEW chunks WITH (full = true)
|
|
```
|
|
|
|
On earlier deployment versions, `full = true` asks for every row to be
|
|
recomputed, and without it a refresh can reconcile only what changed in the
|
|
source. Recent versions refuse `WITH` options; whether their refresh rebuilds
|
|
the whole view, as every local SDK refresh does, or adds only what changed
|
|
depends on the version and on the view's query.
|
|
|
|
<Warning>
|
|
`CASCADE` asks for the views this one is built on to be refreshed first, in
|
|
dependency order. Whether a deployment carries it out depends on its version,
|
|
and recent versions refuse it; where it is refused, refresh each view yourself,
|
|
its sources first.
|
|
</Warning>
|
|
|
|
## Watching a refresh {#watching}
|
|
|
|
Both statements submit work that outlives the call. Track it with
|
|
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
|
[`SHOW ERRORS`](/sql/ddl/job#errors).
|
|
|
|
## See also {#see-also}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
The same operation from the SDKs, and what a refresh guarantees.
|
|
</Card>
|
|
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
|
|
Declaring the columns this fills.
|
|
</Card>
|
|
</CardGroup>
|