feat: add materialized view refresh resource budgets (#4902)

Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.

```python
job = view.refresh_async(
    cpu_millis=8000,
    memory_bytes=16 * 1024**3,
    gpu_count=2,
    gpu_class="h100",
)
job.wait()
```

```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
      gpu_count = 2, gpu_class = 'h100');
```

The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.

Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.

The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.

Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
This commit is contained in:
Jack Ye
2026-11-09 15:02:13 -08:00
committed by GitHub
parent 1eeb200705
commit 794fa023cd
11 changed files with 456 additions and 28 deletions
+2 -9
View File
@@ -240,18 +240,11 @@ jobs:
sudo mkswap "$swapfile"
sudo swapon "$swapfile"
- name: Build
run: cargo build --profile ci --all-features --tests --locked --examples
- name: Reclaim completed workspace build artifacts
run: |
# Package-level tests resolve a different feature graph from the workspace
# build. Reclaim its lancedb outputs before linking the test graph.
df -h .
cargo clean --profile ci -p lancedb
df -h .
run: cargo build --profile ci -p lancedb --all-features --tests --locked --examples
- name: Run feature tests
run: CARGO_ARGS="--profile ci" make -C ./lancedb feature-tests
- name: Run examples
run: cargo run --profile ci --all-features --example simple --locked
run: cargo run --profile ci -p lancedb --all-features --example simple --locked
- name: Run blob v2 docs example
# Python and TypeScript blob docs tests install preview packages.
# Those builds are not guaranteed to remain available for more than