mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 23:32:27 +00:00
Brendan Clement
bbadf0dbea
The SDK now sends CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW to a db:// deployment as SQL, and refuses source_version on a remote refresh. Current Sophon source accepts both statements, with CASCADE and WITH (full = true), but only on deployments that enable materialized views, and whether a refresh rebuilds the view or reconciles changes depends on configuration. The pages said there is no CREATE MATERIALIZED VIEW statement, that refreshes are incremental by default and that CASCADE is not implemented. The refresh, management and SQL reference pages now describe what the SDK sends, say that materialized views are experimental and must be enabled on a deployment, and leave refresh behaviour, CASCADE and the job type to the deployment's version and configuration. The SQL and management summaries list CREATE MATERIALIZED VIEW. No deployment was exercised.
113 lines
4.6 KiB
Plaintext
113 lines
4.6 KiB
Plaintext
---
|
|
title: "Refresh a materialized view"
|
|
sidebarTitle: "Materialized View Refresh"
|
|
description: "A lakehouse object: a table defined by a query over another table, recomputed by each refresh."
|
|
icon: table
|
|
keywords: ["materialized view", "create_materialized_view", "refresh", "full refresh"]
|
|
---
|
|
|
|
import {
|
|
PyMaterializedViewCreate as MaterializedViewCreate,
|
|
PyMaterializedViewRefresh as MaterializedViewRefresh,
|
|
} from '/snippets/materialized_view.mdx';
|
|
|
|
A materialized view is a lakehouse object: a table whose contents are defined by
|
|
a query over another table. The query is recorded in the view's schema;
|
|
creating the view computes its rows, and a refresh computes them again.
|
|
|
|
The result is an ordinary table. It can be queried, indexed, and searched, and
|
|
it appears in `table_names()` — nothing about reading it is special.
|
|
|
|
<Note>
|
|
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view).
|
|
</Note>
|
|
|
|
## Create and refresh {#create-and-refresh}
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{MaterializedViewCreate}
|
|
</CodeBlock>
|
|
|
|
Creating the view computes its rows, so it is ready to read. Pass
|
|
`with_no_data=True` to create it empty instead. Later changes to the source
|
|
reach the view through a refresh:
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{MaterializedViewRefresh}
|
|
</CodeBlock>
|
|
|
|
<Note>
|
|
These results were verified with the development SDK 0.40.0b13, built from
|
|
lancedb main. The published 0.39.0 differs: the source must be created with
|
|
stable row ids (the `new_table_enable_stable_row_ids` storage option),
|
|
`create_materialized_view` creates the view empty and has no `with_no_data`,
|
|
and `refresh()` still accepts `full`. Its first `refresh()` writes every
|
|
matching row, two in this example.
|
|
</Note>
|
|
|
|
`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
|
|
Omitting it selects every source column, expanded against the schema as it was
|
|
at creation time. `where` filters source rows; `limit` caps the view.
|
|
|
|
## Any table can be the source {#stable-row-ids}
|
|
|
|
The source needs no special storage option. A refresh does not track which
|
|
source rows the view came from, so stable row ids
|
|
(`new_table_enable_stable_row_ids`) are not required: the example above creates
|
|
its source with the defaults. Earlier development builds, and the published
|
|
0.39.0, required them.
|
|
|
|
## Every refresh rebuilds the view {#incremental}
|
|
|
|
`refresh()` runs the view's query against the source and replaces everything
|
|
the view holds with the result, in one commit. It does not reconcile changes:
|
|
`rows_written` is the size of the whole result — two after adding Bea, not
|
|
one — and a refresh with nothing new to read still rewrites the view and
|
|
commits a new table version.
|
|
|
|
`source_version=N` refreshes against that source version instead of the latest,
|
|
which is how a view reproduces a known point in time. There is no `full`
|
|
argument; every refresh is a full one.
|
|
|
|
## Against a deployment {#availability}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
`create_materialized_view` and `refresh` also work on a `db://` connection.
|
|
There the SDK sends them to the deployment as SQL -- `CREATE MATERIALIZED VIEW`
|
|
and `REFRESH MATERIALIZED VIEW` -- and the work runs on the cluster rather than
|
|
in your process. `source_version` is for local views only; the SDK refuses it
|
|
on a remote refresh.
|
|
|
|
Materialized views are experimental on a deployment, which has to have them
|
|
enabled. How its refresh works -- rebuilding the whole view, as above, or
|
|
reconciling what changed in the source -- depends on its LanceDB Enterprise
|
|
version and configuration, and so do the options
|
|
[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view)
|
|
accepts.
|
|
|
|
`SHOW MATERIALIZED VIEWS` lists them, which is cheaper than
|
|
`list_materialized_views` -- that method finds views by reading every table's
|
|
schema, so it costs one open per table.
|
|
|
|
## Working with the view {#working-with-the-view}
|
|
|
|
`view.table` is the underlying table, for querying, indexing, or search.
|
|
`view.definition` is the recorded query — the select list, predicate, and
|
|
limit — which is how you check what a view actually contains without inferring
|
|
it from the rows. Both are properties; on an async view, the definition is read
|
|
with `await view.definition()`.
|
|
|
|
An existing view is reopened with `db.open_materialized_view(name)`.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Derive a column instead of a whole table.
|
|
</Card>
|
|
<Card title="Versioning" icon="clock-rotate-left" href="/management/table#versioning">
|
|
Every refresh commits a table version.
|
|
</Card>
|
|
</CardGroup>
|