mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
feat: add materialized view refresh resource budgets (#4902)
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.
```python
job = view.refresh_async(
cpu_millis=8000,
memory_bytes=16 * 1024**3,
gpu_count=2,
gpu_class="h100",
)
job.wait()
```
```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
gpu_count = 2, gpu_class = 'h100');
```
The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.
Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.
The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.
Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
This commit is contained in:
@@ -240,18 +240,11 @@ jobs:
|
||||
sudo mkswap "$swapfile"
|
||||
sudo swapon "$swapfile"
|
||||
- name: Build
|
||||
run: cargo build --profile ci --all-features --tests --locked --examples
|
||||
- name: Reclaim completed workspace build artifacts
|
||||
run: |
|
||||
# Package-level tests resolve a different feature graph from the workspace
|
||||
# build. Reclaim its lancedb outputs before linking the test graph.
|
||||
df -h .
|
||||
cargo clean --profile ci -p lancedb
|
||||
df -h .
|
||||
run: cargo build --profile ci -p lancedb --all-features --tests --locked --examples
|
||||
- name: Run feature tests
|
||||
run: CARGO_ARGS="--profile ci" make -C ./lancedb feature-tests
|
||||
- name: Run examples
|
||||
run: cargo run --profile ci --all-features --example simple --locked
|
||||
run: cargo run --profile ci -p lancedb --all-features --example simple --locked
|
||||
- name: Run blob v2 docs example
|
||||
# Python and TypeScript blob docs tests install preview packages.
|
||||
# Those builds are not guaranteed to remain available for more than
|
||||
|
||||
Reference in New Issue
Block a user