rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-14 22:56:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add materialized view refresh resource budgets (#4902)

Browse Source 
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.

```python
job = view.refresh_async(
    cpu_millis=8000,
    memory_bytes=16 * 1024**3,
    gpu_count=2,
    gpu_class="h100",
)
job.wait()
```

```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
      gpu_count = 2, gpu_class = 'h100');
```

The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.

Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.

The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.

Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
This commit is contained in:
Jack Ye
2026-11-09 15:02:13 -08:00
committed by GitHub
parent 1eeb200705
commit 794fa023cd
11 changed files with 456 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/sql/dml/refresh-materialized-view.mdx
+35 -1
View File
@@ -14,7 +14,8 @@ A LanceDB extension with no DataFusion equivalent.
<CodeGroup>
```sql SQL icon="database"
REFRESH MATERIALIZED VIEW <name>
[WITH (full = <boolean>)]
[WITH (full = <boolean>, cpu_millis = <integer>, memory_bytes = <integer>,
gpu_count = <integer>, gpu_class = '<class>')]
```
</CodeGroup>
@@ -36,6 +37,39 @@ regardless of what a local SDK's `refresh()` method accepts -- do not
assume the two option sets match.
</Note>
## Resource budgets {#resources}
Set budgets for one refresh execution:
```sql
REFRESH MATERIALIZED VIEW curated_training_set WITH (
cpu_millis = 8000,
memory_bytes = 17179869184,
gpu_count = 2,
gpu_class = 'h100'
);
```
`cpu_millis` is CPU capacity in millicores (1000 = one core), not CPU time.
`memory_bytes` caps memory in bytes. `gpu_count` caps concurrent GPUs for the
refresh. These are aggregate caps across its SQL execution and Function
processes; they are not multiplied by the number of Functions in the query.
CPU or memory requires all three budget fields. CPU and memory must be positive;
`gpu_count = 0` is allowed for CPU-only views. A GPU budget alone is also accepted.
A cap does not reserve all of its GPUs or increase refresh parallelism. Current
MV refreshes use at most one GPU, shared by their GPU Function calls.
`gpu_class` selects an operator-configured class for this execution, leaving the
Function registration and saved MV placement unchanged. Use `SHOW GPU CLASSES`
to discover configured classes. Omitted budgets and class preserve defaults.
Options can be combined with `full = true`.
This requires an Enterprise server and an isolated execution backend that can
enforce resource budgets. Unsupported backends reject constrained refreshes.
Older servers reject the new options. Saved defaults and automatic refresh
configuration are unchanged.
## Watching a refresh {#watching}
A refresh submits work that outlives the call. Track it with