mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-29 14:56:51 +00:00
Xuanwo
8a4effa06a
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files.
125 lines
5.4 KiB
Plaintext
125 lines
5.4 KiB
Plaintext
---
|
|
title: "Refresh a materialized view"
|
|
sidebarTitle: "Materialized View Refresh"
|
|
description: "Create a materialized view and refresh it when the source data changes."
|
|
icon: table
|
|
keywords: ["materialized view", "create_materialized_view", "refresh", "full refresh"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Local creation and refresh require a source build with the experimental-functions Cargo feature. Enterprise runs the work on remote compute. See [Enterprise refreshes](#availability) for server compatibility requirements.
|
|
---
|
|
|
|
import {
|
|
PyMaterializedViewCreate as MaterializedViewCreate,
|
|
PyMaterializedViewRefresh as MaterializedViewRefresh,
|
|
} from '/snippets/materialized_view.mdx';
|
|
|
|
A materialized view stores the result of a query as a table. Use it to keep a
|
|
filtered or transformed dataset that you can query, index and search repeatedly.
|
|
Refresh the view after its source data changes.
|
|
|
|
<Note>
|
|
In the 0.41 SDK release, local materialized-view creation and refresh require a
|
|
source build with the `experimental-functions` Cargo feature. Published Python
|
|
and TypeScript packages leave it disabled. Opening, reading, listing and dropping
|
|
existing views remain available, as do remote Cloud and Enterprise operations.
|
|
</Note>
|
|
|
|
<Note>
|
|
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view).
|
|
</Note>
|
|
|
|
## Create and refresh {#create-and-refresh}
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{MaterializedViewCreate}
|
|
</CodeBlock>
|
|
|
|
Creating the view computes its rows, so it is ready to read. Pass
|
|
`with_no_data=True` to create it empty instead. Later changes to the source
|
|
reach the view through a refresh:
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{MaterializedViewRefresh}
|
|
</CodeBlock>
|
|
|
|
The example works with Python SDK 0.40.0. It creates the source with stable
|
|
row IDs, which that release requires for ordinary materialized-view sources.
|
|
|
|
`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
|
|
Omitting it selects every source column, expanded against the schema as it was
|
|
at creation time. `where` filters source rows; `limit` caps the view.
|
|
|
|
## Prepare the source table {#stable-row-ids}
|
|
|
|
In Python SDK 0.40.0, create the source with
|
|
`storage_options={"new_table_enable_stable_row_ids": "true"}`, as the example
|
|
above does. Stable row IDs let the view track source rows across updates and
|
|
compaction. Set this option when creating the table; it cannot be added later.
|
|
|
|
A source with the [realtime engine](/tables/realtime-engine) enabled or retained, uncompacted write-ahead-log
|
|
rows is not accepted by the local materialized-view implementation.
|
|
|
|
## Refresh behavior {#incremental}
|
|
|
|
With Python SDK 0.40.0, a refresh applies source changes incrementally when the
|
|
query supports it, and rebuilds the view otherwise. In the example, adding Bea
|
|
writes one new view row. Refreshing again without source changes writes none.
|
|
Use `refresh(full=True)` to force a rebuild.
|
|
|
|
`rows_written` counts rows written by that refresh, not all rows in the view.
|
|
Use `view.table.count_rows()` to get the total. Pass `source_version=N` to
|
|
refresh against a particular source version instead of the latest.
|
|
|
|
<Note>
|
|
Refresh behavior has changed between SDK versions. The earlier 0.40.0b13
|
|
development build rebuilt every local view on refresh and did not accept
|
|
`full`; 0.39.0 created views empty. Use the reference for your installed SDK
|
|
when maintaining an existing pipeline.
|
|
</Note>
|
|
|
|
## Enterprise refreshes {#availability}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
Enterprise runs materialized-view creation and refresh on remote compute.
|
|
For SQL, see [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view) and
|
|
[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view).
|
|
|
|
Use an SDK version supported by your Enterprise deployment. Materialized-view
|
|
APIs and stored definitions have changed across releases; a local SDK example
|
|
alone does not establish compatibility with an older server. Before upgrading
|
|
an existing pipeline, confirm the supported refresh options and whether its
|
|
views need to be recreated with your administrator.
|
|
|
|
Stable row IDs and feature configuration for Enterprise sources are managed on
|
|
the server. The source-table storage option in the local example does not
|
|
configure the Enterprise deployment, and Python SDK 0.40.0's `create_table` on a
|
|
`db://` connection does not accept `storage_options`: leave it out, and confirm
|
|
with your administrator that the deployment gives new tables stable row IDs.
|
|
|
|
`list_materialized_views` requests the server's view listing. On a local
|
|
database it reads table schemas to find views.
|
|
|
|
## Working with the view {#working-with-the-view}
|
|
|
|
`view.table` is the underlying table, for querying, indexing, or search.
|
|
`view.definition` is the recorded query: the select list, predicate, and
|
|
limit. Use it to inspect the query without inferring it from the rows. Both are properties; on an async view, the definition is read
|
|
with `await view.definition()`.
|
|
|
|
An existing view is reopened with `db.open_materialized_view(name)`.
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Derive a column instead of a whole table.
|
|
</Card>
|
|
<Card title="Versioning" icon="clock-rotate-left" href="/management/table#versioning">
|
|
Inspect the table history and read a particular version.
|
|
</Card>
|
|
</CardGroup>
|