mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 15:52:26 +00:00
Brendan Clement
bbadf0dbea
The SDK now sends CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW to a db:// deployment as SQL, and refuses source_version on a remote refresh. Current Sophon source accepts both statements, with CASCADE and WITH (full = true), but only on deployments that enable materialized views, and whether a refresh rebuilds the view or reconciles changes depends on configuration. The pages said there is no CREATE MATERIALIZED VIEW statement, that refreshes are incremental by default and that CASCADE is not implemented. The refresh, management and SQL reference pages now describe what the SDK sends, say that materialized views are experimental and must be enabled on a deployment, and leave refresh behaviour, CASCADE and the job type to the deployment's version and configuration. The SQL and management summaries list CREATE MATERIALIZED VIEW. No deployment was exercised.
99 lines
3.2 KiB
Plaintext
99 lines
3.2 KiB
Plaintext
---
|
|
title: "REFRESH"
|
|
sidebarTitle: "REFRESH"
|
|
description: "Filling computed columns and bringing materialized views up to date."
|
|
icon: arrows-rotate
|
|
keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
|
|
|
LanceDB extensions with no DataFusion equivalent. They are what makes
|
|
[feature engineering](/feature-engineering) reachable from SQL: a column
|
|
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
|
|
|
|
## REFRESH COLUMN {#refresh-column}
|
|
|
|
```sql
|
|
REFRESH COLUMN <column> ON [TABLE] <table>
|
|
[WHERE <predicate>]
|
|
[WITH (<option> = <value> [, ...])]
|
|
```
|
|
|
|
```sql
|
|
REFRESH COLUMN doubled ON my_table
|
|
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
|
```
|
|
|
|
A refresh gives values to rows that lack them, recomputes rows whose inputs
|
|
changed, and **leaves other filled rows alone**. That makes it idempotent, cheap
|
|
to repeat as data arrives, and safe to re-run after a failure -- though a re-run
|
|
can recompute values the failed run had already written.
|
|
|
|
`WHERE` scopes the refresh, which is how you fill a slice and check it before
|
|
committing to the whole table.
|
|
|
|
### Options {#refresh-column-options}
|
|
|
|
| Option | Meaning |
|
|
| --- | --- |
|
|
| `num_workers` | how many workers to use |
|
|
| `cpu_millis` | CPU budget per worker |
|
|
| `memory_bytes` | memory budget per worker |
|
|
| `gpu_count` | GPUs per worker |
|
|
| `batch_size` | rows a worker takes at a time |
|
|
| `error_policy` | `'skip'` continues past failing rows instead of failing the job |
|
|
|
|
```sql
|
|
REFRESH COLUMN embedding ON documents
|
|
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
|
|
```
|
|
|
|
`num_workers` sets a count; the budget options size each worker. They are
|
|
alternatives rather than a pair.
|
|
|
|
<Note>
|
|
`error_policy` and `num_workers` are refused for a column filled by a registered
|
|
[function](/management/function) with its own resource budget -- use the CPU,
|
|
memory and GPU options there.
|
|
</Note>
|
|
|
|
## REFRESH MATERIALIZED VIEW {#refresh-materialized-view}
|
|
|
|
```sql
|
|
REFRESH MATERIALIZED VIEW <name> [CASCADE] [WITH (<option> = <value>)]
|
|
```
|
|
|
|
```sql
|
|
REFRESH MATERIALIZED VIEW chunks
|
|
REFRESH MATERIALIZED VIEW chunks WITH (full = true)
|
|
```
|
|
|
|
`full = true` asks for every row to be recomputed. Without it, whether a
|
|
refresh rebuilds the whole view -- as every local SDK refresh does -- or
|
|
reconciles only what changed in the source depends on the deployment's LanceDB
|
|
Enterprise version and configuration.
|
|
|
|
<Warning>
|
|
`CASCADE` asks for the views this one is built on to be refreshed first, in
|
|
dependency order. Whether a deployment carries it out depends on its version;
|
|
where it does not, refresh each view yourself, its sources first.
|
|
</Warning>
|
|
|
|
## Watching a refresh {#watching}
|
|
|
|
Both statements submit work that outlives the call. Track it with
|
|
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
|
[`SHOW ERRORS`](/sql/ddl/job#errors).
|
|
|
|
## See also {#see-also}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
The same operation from the SDKs, and what a refresh guarantees.
|
|
</Card>
|
|
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
|
|
Declaring the columns this fills.
|
|
</Card>
|
|
</CardGroup>
|