rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-13 14:16:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add materialized view refresh resource budgets (#4902)

Browse Source 
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.

```python
job = view.refresh_async(
    cpu_millis=8000,
    memory_bytes=16 * 1024**3,
    gpu_count=2,
    gpu_class="h100",
)
job.wait()
```

```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
      gpu_count = 2, gpu_class = 'h100');
```

The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.

Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.

The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.

Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
This commit is contained in:
Jack Ye
2026-11-09 15:02:13 -08:00
committed by GitHub
parent 1eeb200705
commit 794fa023cd
11 changed files with 456 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/src/table.rs
+36 -2
View File
@@ -2309,11 +2309,16 @@ impl Table {
})
}
#[pyo3(signature = (source_version=None, *, full=false))]
#[pyo3(signature = (source_version=None, *, full=false, cpu_millis=None, memory_bytes=None, gpu_count=None, gpu_class=None))]
#[allow(clippy::too_many_arguments)]
pub fn refresh_materialized_view(
self_: PyRef<'_, Self>,
source_version: Option<u64>,
full: bool,
cpu_millis: Option<u64>,
memory_bytes: Option<u64>,
gpu_count: Option<u32>,
gpu_class: Option<String>,
) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move {
@@ -2321,6 +2326,18 @@ impl Table {
.await
.infer_error()?;
let mut builder = view.refresh().full(full);
if let Some(value) = cpu_millis {
builder = builder.cpu_millis(value);
}
if let Some(value) = memory_bytes {
builder = builder.memory_bytes(value);
}
if let Some(value) = gpu_count {
builder = builder.gpu_count(value);
}
if let Some(value) = gpu_class {
builder = builder.gpu_class(value);
}
if let Some(version) = source_version {
builder = builder.source_version(version);
}
@@ -2329,11 +2346,16 @@ impl Table {
})
}
#[pyo3(signature = (source_version=None, *, full=false))]
#[pyo3(signature = (source_version=None, *, full=false, cpu_millis=None, memory_bytes=None, gpu_count=None, gpu_class=None))]
#[allow(clippy::too_many_arguments)]
pub fn refresh_materialized_view_async(
self_: PyRef<'_, Self>,
source_version: Option<u64>,
full: bool,
cpu_millis: Option<u64>,
memory_bytes: Option<u64>,
gpu_count: Option<u32>,
gpu_class: Option<String>,
) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move {
@@ -2341,6 +2363,18 @@ impl Table {
.await
.infer_error()?;
let mut builder = view.refresh().full(full);
if let Some(value) = cpu_millis {
builder = builder.cpu_millis(value);
}
if let Some(value) = memory_bytes {
builder = builder.memory_bytes(value);
}
if let Some(value) = gpu_count {
builder = builder.gpu_count(value);
}
if let Some(value) = gpu_class {
builder = builder.gpu_class(value);
}
if let Some(version) = source_version {
builder = builder.source_version(version);
}