mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-22 11:42:27 +00:00
Adds a user-facing realtime engine (RTE) page at `docs/web/tables/realtime-engine.mdx`, listed under **Management** after Branches. It is adapted from the internal WAL field guide (lancedb/sophon#6213), with the deployment and operations material left out. The page covers: - what RTE is and how it works (memtable → SSTables → compaction; reads merge the base table with the fresh tier) - getting started: `set_unenforced_primary_key`, indexes, `enable_rte` / `ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial updates, deletes, and `checkpoint_rte` / `disable_rte` - performance: write throughput and shard scaling, and read latency with and without RTE - a note on operations not yet supported (time travel, branching, `count_rows`, row IDs, `TABLESAMPLE`) Also aligns related docs with current RTE naming and behavior: - `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` / `UNSET RTE` - `maintenance/background-jobs.mdx`: job names are `memwal_compaction` and `memwal_catchup_<type>` - `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine - Python `AsyncTable.enable_rte` / `RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes created after enabling RTE are maintained too The examples are written directly in the page rather than generated from `docs/web-tests`, because RTE needs an Enterprise server to run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: XY Zhan <zhaner08@hotmail.com>
118 lines
5.0 KiB
Plaintext
118 lines
5.0 KiB
Plaintext
---
|
|
title: "Refresh a materialized view"
|
|
sidebarTitle: "Materialized View Refresh"
|
|
description: "Create a materialized view and refresh it when the source data changes."
|
|
icon: table
|
|
keywords: ["materialized view", "create_materialized_view", "refresh", "full refresh"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Both offerings support materialized views. OSS refreshes run in your process; Enterprise runs the work on remote compute. See [Enterprise refreshes](#availability) for server compatibility requirements.
|
|
---
|
|
|
|
import {
|
|
PyMaterializedViewCreate as MaterializedViewCreate,
|
|
PyMaterializedViewRefresh as MaterializedViewRefresh,
|
|
} from '/snippets/materialized_view.mdx';
|
|
|
|
A materialized view stores the result of a query as a table. Use it to keep a
|
|
filtered or transformed dataset that you can query, index and search repeatedly.
|
|
Refresh the view after its source data changes.
|
|
|
|
<Note>
|
|
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view).
|
|
</Note>
|
|
|
|
## Create and refresh {#create-and-refresh}
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{MaterializedViewCreate}
|
|
</CodeBlock>
|
|
|
|
Creating the view computes its rows, so it is ready to read. Pass
|
|
`with_no_data=True` to create it empty instead. Later changes to the source
|
|
reach the view through a refresh:
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{MaterializedViewRefresh}
|
|
</CodeBlock>
|
|
|
|
The example works with Python SDK 0.40.0. It creates the source with stable
|
|
row IDs, which that release requires for ordinary materialized-view sources.
|
|
|
|
`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
|
|
Omitting it selects every source column, expanded against the schema as it was
|
|
at creation time. `where` filters source rows; `limit` caps the view.
|
|
|
|
## Prepare the source table {#stable-row-ids}
|
|
|
|
In Python SDK 0.40.0, create the source with
|
|
`storage_options={"new_table_enable_stable_row_ids": "true"}`, as the example
|
|
above does. Stable row IDs let the view track source rows across updates and
|
|
compaction. Set this option when creating the table; it cannot be added later.
|
|
|
|
A source with the [realtime engine](/tables/realtime-engine) enabled or retained, uncompacted write-ahead-log
|
|
rows is not accepted by the local materialized-view implementation.
|
|
|
|
## Refresh behavior {#incremental}
|
|
|
|
With Python SDK 0.40.0, a refresh applies source changes incrementally when the
|
|
query supports it, and rebuilds the view otherwise. In the example, adding Bea
|
|
writes one new view row. Refreshing again without source changes writes none.
|
|
Use `refresh(full=True)` to force a rebuild.
|
|
|
|
`rows_written` counts rows written by that refresh, not all rows in the view.
|
|
Use `view.table.count_rows()` to get the total. Pass `source_version=N` to
|
|
refresh against a particular source version instead of the latest.
|
|
|
|
<Note>
|
|
Refresh behavior has changed between SDK versions. The earlier 0.40.0b13
|
|
development build rebuilt every local view on refresh and did not accept
|
|
`full`; 0.39.0 created views empty. Use the reference for your installed SDK
|
|
when maintaining an existing pipeline.
|
|
</Note>
|
|
|
|
## Enterprise refreshes {#availability}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
Enterprise runs materialized-view creation and refresh on remote compute.
|
|
For SQL, see [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view) and
|
|
[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view).
|
|
|
|
Use an SDK version supported by your Enterprise deployment. Materialized-view
|
|
APIs and stored definitions have changed across releases; a local SDK example
|
|
alone does not establish compatibility with an older server. Before upgrading
|
|
an existing pipeline, confirm the supported refresh options and whether its
|
|
views need to be recreated with your administrator.
|
|
|
|
Stable row IDs and feature configuration for Enterprise sources are managed on
|
|
the server. The source-table storage option in the local example does not
|
|
configure the Enterprise deployment, and Python SDK 0.40.0's `create_table` on a
|
|
`db://` connection does not accept `storage_options`: leave it out, and confirm
|
|
with your administrator that the deployment gives new tables stable row IDs.
|
|
|
|
`list_materialized_views` requests the server's view listing. On a local
|
|
database it reads table schemas to find views.
|
|
|
|
## Working with the view {#working-with-the-view}
|
|
|
|
`view.table` is the underlying table, for querying, indexing, or search.
|
|
`view.definition` is the recorded query: the select list, predicate, and
|
|
limit. Use it to inspect the query without inferring it from the rows. Both are properties; on an async view, the definition is read
|
|
with `await view.definition()`.
|
|
|
|
An existing view is reopened with `db.open_materialized_view(name)`.
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Derive a column instead of a whole table.
|
|
</Card>
|
|
<Card title="Versioning" icon="clock-rotate-left" href="/management/table#versioning">
|
|
Inspect the table history and read a particular version.
|
|
</Card>
|
|
</CardGroup>
|