mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Jack Ye
5b5ed930ac
Every group now answers exactly one question. Get started and Concepts cover what the system is; the verb groups cover doing something with it; Operate, Secure and Deploy cover running it; SQL reference is lookup. Maintenance becomes Operate, Governance becomes Secure, and the rest take verbs to match. Three new pages in Concepts and Get started: Guarantees states what a write promises when it returns, which version a reader sees, what survives a failure, and -- the half usually missing -- what is not guaranteed: no multi-table transactions, no read-your-writes across processes without an interval, unenforced primary keys that do not enforce, no background compaction. Limits collects bounds that were stated in passing across a dozen pages, with the numbers verified against the code rather than inferred: name lengths, job event caps, SHOW ERRORS bounds, the sampling range that is refused rather than clamped. Architecture illustrates embedded against distributed as the one deployment decision, and is written to cover both rather than only Enterprise. Indexing splits by question. Choosing an index is a Query decision and moves there; building, repairing and GPU-building it stay in Operate. SQL becomes a top-level reference of fifteen pages instead of a sub-group of Query. Auditing sophon-sql turned up a large undocumented surface: ALTER TABLE ADD COLUMN in its plain, computed and function forms with FILTER, IGNORE NULLS and OVER modifiers; SET UNENFORCED PRIMARY KEY; SET and UNSET WAL with sharding; CREATE FUNCTION with its dependency options; REFRESH COLUMN and REFRESH MATERIALIZED VIEW; and the SHOW family. None of it was written down.
96 lines
3.0 KiB
Plaintext
96 lines
3.0 KiB
Plaintext
---
|
|
title: "REFRESH"
|
|
sidebarTitle: "Refresh"
|
|
description: "Filling computed columns and bringing materialized views up to date."
|
|
icon: arrows-rotate
|
|
keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
|
|
|
LanceDB extensions with no DataFusion equivalent. They are what makes
|
|
[feature engineering](/feature-engineering) reachable from SQL: a column
|
|
declared by [DDL](/sql/ddl#computed-column) starts empty, and a refresh fills it.
|
|
|
|
## REFRESH COLUMN {#refresh-column}
|
|
|
|
```sql
|
|
REFRESH COLUMN <column> ON [TABLE] <table>
|
|
[WHERE <predicate>]
|
|
[WITH (<option> = <value> [, ...])]
|
|
```
|
|
|
|
```sql
|
|
REFRESH COLUMN doubled ON my_table
|
|
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
|
```
|
|
|
|
A refresh gives values to rows that lack them and **leaves filled rows alone**.
|
|
That makes it idempotent, cheap to repeat as data arrives, and resumable after a
|
|
failure -- re-running costs only the rows that did not complete.
|
|
|
|
`WHERE` scopes the refresh, which is how you fill a slice and check it before
|
|
committing to the whole table.
|
|
|
|
### Options {#refresh-column-options}
|
|
|
|
| Option | Meaning |
|
|
| --- | --- |
|
|
| `num_workers` | how many workers to use |
|
|
| `cpu_millis` | CPU budget per worker |
|
|
| `memory_bytes` | memory budget per worker |
|
|
| `gpu_count` | GPUs per worker |
|
|
| `batch_size` | rows a worker takes at a time |
|
|
| `error_policy` | `'skip'` continues past failing rows instead of failing the job |
|
|
|
|
```sql
|
|
REFRESH COLUMN embedding ON documents
|
|
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
|
|
```
|
|
|
|
`num_workers` sets a count; the budget options size each worker. They are
|
|
alternatives rather than a pair.
|
|
|
|
<Note>
|
|
`error_policy` and `num_workers` are refused for a column filled by a registered
|
|
[function](/management/function) with its own resource budget -- use the CPU,
|
|
memory and GPU options there.
|
|
</Note>
|
|
|
|
## REFRESH MATERIALIZED VIEW {#refresh-materialized-view}
|
|
|
|
```sql
|
|
REFRESH MATERIALIZED VIEW <name> [WITH (<option> = <value>)]
|
|
```
|
|
|
|
```sql
|
|
REFRESH MATERIALIZED VIEW chunks
|
|
REFRESH MATERIALIZED VIEW chunks WITH (full = true)
|
|
```
|
|
|
|
By default a refresh is **incremental** -- it processes what the source has
|
|
added since the last one. `full = true` recomputes every row, which is what you
|
|
want after changing something the view's rows depend on.
|
|
|
|
<Warning>
|
|
`CASCADE` parses but is **not implemented**. A view built on another view is
|
|
refreshed by refreshing each in dependency order yourself.
|
|
</Warning>
|
|
|
|
## Watching a refresh {#watching}
|
|
|
|
Both statements submit work that outlives the call. Track it with
|
|
[`SHOW JOBS`](/sql/show#jobs), and diagnose failures with
|
|
[`SHOW ERRORS`](/sql/show#errors).
|
|
|
|
## See also {#see-also}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
The same operation from the SDKs, and what a refresh guarantees.
|
|
</Card>
|
|
<Card title="DDL" icon="pen-ruler" href="/sql/ddl#columns">
|
|
Declaring the columns this fills.
|
|
</Card>
|
|
</CardGroup>
|