mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack Ye
794fa023cd
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.
```python
job = view.refresh_async(
cpu_millis=8000,
memory_bytes=16 * 1024**3,
gpu_count=2,
gpu_class="h100",
)
job.wait()
```
```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
gpu_count = 2, gpu_class = 'h100');
```
The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.
Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.
The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.
Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
93 lines
3.3 KiB
Plaintext
93 lines
3.3 KiB
Plaintext
---
|
|
title: "REFRESH MATERIALIZED VIEW"
|
|
sidebarTitle: "REFRESH MATERIALIZED VIEW"
|
|
description: "Bringing a materialized view's stored result up to date."
|
|
icon: arrows-rotate
|
|
keywords: ["refresh materialized view", "cascade"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A LanceDB extension with no DataFusion equivalent.
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH MATERIALIZED VIEW <name>
|
|
[WITH (full = <boolean>, cpu_millis = <integer>, memory_bytes = <integer>,
|
|
gpu_count = <integer>, gpu_class = '<class>')]
|
|
```
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH MATERIALIZED VIEW chunks
|
|
REFRESH MATERIALIZED VIEW chunks WITH (full = true)
|
|
```
|
|
</CodeGroup>
|
|
|
|
A refresh updates the view's stored result. Refresh views in dependency order,
|
|
starting with their source views. `full = true` forces a complete rebuild
|
|
even when the sources are unchanged; the default, `false`, refreshes
|
|
incrementally when supported and rebuilds otherwise.
|
|
|
|
<Note>
|
|
`CASCADE` is not accepted by the SQL implementation described here,
|
|
regardless of what a local SDK's `refresh()` method accepts -- do not
|
|
assume the two option sets match.
|
|
</Note>
|
|
|
|
## Resource budgets {#resources}
|
|
|
|
Set budgets for one refresh execution:
|
|
|
|
```sql
|
|
REFRESH MATERIALIZED VIEW curated_training_set WITH (
|
|
cpu_millis = 8000,
|
|
memory_bytes = 17179869184,
|
|
gpu_count = 2,
|
|
gpu_class = 'h100'
|
|
);
|
|
```
|
|
|
|
`cpu_millis` is CPU capacity in millicores (1000 = one core), not CPU time.
|
|
`memory_bytes` caps memory in bytes. `gpu_count` caps concurrent GPUs for the
|
|
refresh. These are aggregate caps across its SQL execution and Function
|
|
processes; they are not multiplied by the number of Functions in the query.
|
|
|
|
CPU or memory requires all three budget fields. CPU and memory must be positive;
|
|
`gpu_count = 0` is allowed for CPU-only views. A GPU budget alone is also accepted.
|
|
A cap does not reserve all of its GPUs or increase refresh parallelism. Current
|
|
MV refreshes use at most one GPU, shared by their GPU Function calls.
|
|
|
|
`gpu_class` selects an operator-configured class for this execution, leaving the
|
|
Function registration and saved MV placement unchanged. Use `SHOW GPU CLASSES`
|
|
to discover configured classes. Omitted budgets and class preserve defaults.
|
|
Options can be combined with `full = true`.
|
|
|
|
This requires an Enterprise server and an isolated execution backend that can
|
|
enforce resource budgets. Unsupported backends reject constrained refreshes.
|
|
Older servers reject the new options. Saved defaults and automatic refresh
|
|
configuration are unchanged.
|
|
|
|
## Watching a refresh {#watching}
|
|
|
|
A refresh submits work that outlives the call. Track it with
|
|
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
|
[`SHOW ERRORS`](/sql/ddl/job#errors). For per-split progress, worker fleet
|
|
detail, and a `DESCRIBE JOB` example, see [Watching a refresh](/sql/dml/refresh#watching).
|
|
|
|
## See also {#see-also}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Materialized View" icon="table" href="/sql/ddl/materialized-view">
|
|
Creating the view this refreshes.
|
|
</Card>
|
|
<Card title="Refresh a materialized view" icon="layer-group" href="/feature-engineering/refresh-materialized-view">
|
|
The same operation from the SDKs, and automatic refresh.
|
|
</Card>
|
|
<Card title="REFRESH COLUMN" icon="arrows-rotate" href="/sql/dml/refresh-column">
|
|
Filling a computed column, instead of a view.
|
|
</Card>
|
|
</CardGroup>
|