mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack Ye
794fa023cd
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.
```python
job = view.refresh_async(
cpu_millis=8000,
memory_bytes=16 * 1024**3,
gpu_count=2,
gpu_class="h100",
)
job.wait()
```
```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
gpu_count = 2, gpu_class = 'h100');
```
The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.
Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.
The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.
Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
148 lines
5.0 KiB
Plaintext
148 lines
5.0 KiB
Plaintext
---
|
|
title: "Refresh a materialized view"
|
|
sidebarTitle: "Refresh Materialized View"
|
|
description: "Bring a materialized view up to date after its source data changes."
|
|
icon: arrows-rotate
|
|
keywords: ["materialized view", "refresh", "auto_refresh_enabled", "auto_refresh_interval_seconds"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Refresh a [materialized view](/feature-engineering/materialized-views) to
|
|
bring its stored rows up to date after the source table changes.
|
|
|
|
<Note>
|
|
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh-materialized-view).
|
|
</Note>
|
|
|
|
## Refresh a view {#refresh}
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
view.refresh()
|
|
```
|
|
</CodeGroup>
|
|
|
|
In LanceDB Embedded, every refresh rebuilds the view completely from its
|
|
source query -- there is no incremental refresh there. Calling it again
|
|
with no source changes still rebuilds and commits a new table version. In
|
|
LanceDB Enterprise, the server chooses the refresh mode instead; pass
|
|
`full=True` to force a full rebuild there too.
|
|
|
|
In LanceDB Embedded, the result reports what the refresh did:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
result = view.refresh()
|
|
result.rows_written # rows in the rebuilt view
|
|
result.source_version # the source table version the view now reflects
|
|
result.version # the view's own new table version
|
|
```
|
|
</CodeGroup>
|
|
|
|
LanceDB Enterprise refreshes run through its SQL job adapter, which doesn't
|
|
expose these metrics: `mode`, `rows_written` and `source_version` are all
|
|
`None` on an Enterprise result, even though the refresh itself ran.
|
|
|
|
Pass `source_version=N` to refresh against a particular source version
|
|
instead of the latest. This is a LanceDB Embedded capability only --
|
|
LanceDB Enterprise's SQL refresh does not support pinning a source version:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
view.refresh(source_version=12) # LanceDB Embedded only
|
|
```
|
|
</CodeGroup>
|
|
|
|
Use `view.table.count_rows()` for the view's row count at any time; on
|
|
LanceDB Embedded it always matches `rows_written` from the most recent
|
|
refresh.
|
|
|
|
{/*
|
|
Draft: publish this section after the Python/Rust resource-budget SDK support
|
|
is released. TypeScript parity is tracked separately in issue #4903.
|
|
|
|
## Resource budgets {#resources}
|
|
|
|
For training-data curation, bound the resources used to recompute a view:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
job = view.refresh_async(
|
|
cpu_millis=8000,
|
|
memory_bytes=16 * 1024**3,
|
|
gpu_count=2,
|
|
gpu_class="h100",
|
|
)
|
|
job.wait()
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
use lancedb::materialized_view::MaterializedView;
|
|
|
|
async fn refresh(view: &MaterializedView) -> lancedb::Result<()> {
|
|
let job = view
|
|
.refresh()
|
|
.cpu_millis(8000)
|
|
.memory_bytes(16_u64 << 30)
|
|
.gpu_count(2)
|
|
.gpu_class("h100")
|
|
.execute_async()
|
|
.await?;
|
|
job.wait().await?;
|
|
Ok(())
|
|
}
|
|
```
|
|
</CodeGroup>
|
|
|
|
The same options are available on `refresh`, asynchronous Python connections,
|
|
and the Rust refresh builder (`cpu_millis`, `memory_bytes`, `gpu_count`, and
|
|
`gpu_class`). CPU is in millicores (8000 means eight cores), memory is bytes,
|
|
and GPU count is an aggregate concurrent cap. CPU or memory requires all three
|
|
budget fields. Use zero GPUs for a CPU-only view, or specify a GPU cap alone.
|
|
|
|
The cap covers the refresh execution and its Function processes together.
|
|
It does not allocate that budget separately to each Function. A GPU class
|
|
selects placement for this execution; discover configured classes with
|
|
`db.list_gpu_classes()`. A cap does not promise more parallelism: current MV
|
|
refreshes use at most one GPU. Omitted settings keep existing defaults.
|
|
|
|
These controls require an Enterprise server with an isolated backend that can
|
|
enforce them; Embedded and unsupported execution backends reject them. Earlier
|
|
Enterprise servers reject the new SQL options. See the
|
|
[SQL resource options](/sql/dml/refresh-materialized-view#resources).
|
|
|
|
TypeScript does not expose these options yet; SDK parity is tracked in
|
|
[issue #4903](https://github.com/lancedb/lancedb/issues/4903).
|
|
|
|
*/}
|
|
|
|
## Automatic refresh {#auto-refresh}
|
|
|
|
A materialized view can refresh itself on a schedule instead of waiting
|
|
for an explicit `refresh()` call. See [Auto Refresh](/enterprise/auto-refresh)
|
|
for enabling it at creation, the SQL and SDK forms for changing the
|
|
setting afterward, and how it differs from a computed column's
|
|
per-output granularity.
|
|
|
|
## When a refresh fails {#failures}
|
|
|
|
A refresh replaces the view's stored rows in one commit, so a failure leaves
|
|
the view as it was after its last successful refresh. Fix the cause and
|
|
refresh again.
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Create a materialized view" icon="table" href="/feature-engineering/materialized-views">
|
|
Define the query a view stores.
|
|
</Card>
|
|
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
The same idea, for a single derived column.
|
|
</Card>
|
|
<Card title="Auto Refresh" icon="clock-rotate-left" href="/enterprise/auto-refresh">
|
|
Refresh on a schedule instead of calling this yourself.
|
|
</Card>
|
|
</CardGroup>
|