mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
Jack Ye
cb45b67c4c
Indexing loses its wrapper group -- Overview, Vector Index, FTS Index, Scalar Index and Build & Repair sit directly in Maintenance now, followed by Compaction, Cleanup and Background Jobs at the same level. Cleanup is new, split out of Compaction: pruning old versions is a distinct Enterprise job (cleanup) from merging files (compact), even though optimize() in OSS bundles both into one call. The retention, disk-usage and failed-transaction material that used to live under Compaction's umbrella moves with it. Background Jobs is new and covers what nothing else does: the job types a deployment runs on its own -- WAL/SPFresh maintenance, cache prewarming, registry archiving and metering export, and the object-teardown job that follows a namespace or view drop. Every entry is grounded against the job-coordinator's public job_type constants, not the internal two-letter wire codes used for registry storage. Five existing pages (Build & Repair, Table, Function, Column Refresh, Materialized View Refresh) gain a one-line job-type note rather than duplicating a page for job types they already document. Storage folds into Deployment: storage/index (choosing a backend) and storage/configuration (configuring one) merge into a single Storage page -- they never overlapped, one is the decision and one is the how-to -- and move under Deployment along with Monitoring. Cloud Providers loses its nesting; its two pages plus Azure sit flat, and the former Overview is retitled Deployment Models so it doesn't collide with the group's own Overview entry. Redirects cover every moved URL.
126 lines
4.2 KiB
Plaintext
126 lines
4.2 KiB
Plaintext
---
|
|
title: "Refresh a materialized view"
|
|
sidebarTitle: "Materialized View Refresh"
|
|
description: "A lakehouse object: a table defined by a query over another table, kept current incrementally."
|
|
icon: table
|
|
keywords: ["materialized view", "create_materialized_view", "refresh", "incremental"]
|
|
---
|
|
|
|
A materialized view is a lakehouse object: a table whose contents are defined by
|
|
a query over another table. The query is recorded in the view's schema; a
|
|
refresh computes the rows.
|
|
|
|
The result is an ordinary table. It can be queried, indexed, and searched, and
|
|
it appears in `table_names()` — nothing about reading it is special.
|
|
|
|
<Note>
|
|
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view).
|
|
</Note>
|
|
|
|
## Create and refresh {#create-and-refresh}
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
db = lancedb.connect(
|
|
"./.lancedb",
|
|
storage_options={"new_table_enable_stable_row_ids": "true"},
|
|
)
|
|
table = db.create_table("people", [{"name": "ada", "age": 36},
|
|
{"name": "kid", "age": 7}])
|
|
|
|
view = db.create_materialized_view(
|
|
"adults",
|
|
"people",
|
|
select=["name", ("shout", "upper(name)")],
|
|
where="age >= 18",
|
|
)
|
|
|
|
result = view.refresh()
|
|
result.rows_written
|
|
# 1
|
|
```
|
|
|
|
`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
|
|
Omitting it selects every source column, expanded against the schema as it was
|
|
at creation time. `where` filters source rows; `limit` caps the view.
|
|
|
|
## The source needs stable row ids {#stable-row-ids}
|
|
|
|
<Warning>
|
|
The source table must have been created with stable row ids:
|
|
|
|
```python
|
|
db = lancedb.connect(uri, storage_options={"new_table_enable_stable_row_ids": "true"})
|
|
```
|
|
|
|
They are what keeps the view's provenance valid when the source is compacted,
|
|
and **they cannot be enabled after a table exists**. A table created without
|
|
them cannot later become the source of a view — it has to be recreated.
|
|
</Warning>
|
|
|
|
This is the one decision here that is hard to undo, so it is worth setting the
|
|
option on any table you might later want to derive from.
|
|
|
|
## Refreshes are incremental {#incremental}
|
|
|
|
`refresh()` reconciles the source's changes into the view — rows added, changed,
|
|
or removed since the last refresh — rather than recomputing from scratch. It
|
|
falls back to a full rebuild when the changes cannot be reconciled.
|
|
|
|
Two arguments override that:
|
|
|
|
- `full=True` forces a rebuild.
|
|
- `source_version=N` refreshes against that source version instead of the
|
|
latest, which is how you make a view reproduce a known point in time.
|
|
|
|
Concurrent refreshes of one view do not duplicate rows. If two plan the same
|
|
source rows, they conflict on commit and the loser raises rather than writing
|
|
them twice — so a retry loop around `refresh()` is safe.
|
|
|
|
<Note>
|
|
**Enterprise job type:** `mat_view_refresh`.
|
|
</Note>
|
|
|
|
## Against a deployment {#availability}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
`create_materialized_view` works on a `db://` connection as well as a local one.
|
|
What changes is where the refresh runs -- across the cluster rather than in your
|
|
process -- and that SQL can drive it:
|
|
|
|
```sql
|
|
REFRESH MATERIALIZED VIEW chunks
|
|
```
|
|
|
|
`WITH (full = true)` recomputes every row instead of only the rows the source
|
|
has added since the last refresh. `SHOW MATERIALIZED VIEWS` lists them, which is
|
|
cheaper than `list_materialized_views` -- that method finds views by reading
|
|
every table's schema, so it costs one open per table.
|
|
|
|
<Note>
|
|
`CASCADE` is accepted by the parser but not yet implemented. A view built on
|
|
another view is refreshed by refreshing each in dependency order.
|
|
</Note>
|
|
|
|
## Working with the view {#working-with-the-view}
|
|
|
|
`view.table()` returns the underlying table for querying, indexing, or search.
|
|
`view.definition()` returns the recorded query — the select list, predicate, and
|
|
limit — which is how you check what a view actually contains without inferring
|
|
it from the rows.
|
|
|
|
An existing view is reopened with `db.open_materialized_view(name)`.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Derive a column instead of a whole table.
|
|
</Card>
|
|
<Card title="Versioning" icon="clock-rotate-left" href="/management/table#versioning">
|
|
Every refresh commits a table version.
|
|
</Card>
|
|
</CardGroup>
|