mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Jack Ye
65ed41333b
Both the lancedb client and the geneva client expose `create_materialized_view`, and they are not the same feature. The page said materialized views are local only, which is true of this client and misleading for anyone on Enterprise, where views are created through geneva and refreshed on the cluster. Found by assembling the combined site and reading the Geneva pages against the new ones: they described the same call as distributed and Enterprise while this one described it as raising. Both were accurate about their own client and neither said which client it meant.
114 lines
4.1 KiB
Plaintext
114 lines
4.1 KiB
Plaintext
---
|
|
title: "Materialized views"
|
|
sidebarTitle: "Materialized views"
|
|
description: "Derive a table from a query over another table, and keep it current incrementally."
|
|
icon: table
|
|
keywords: ["materialized view", "create_materialized_view", "refresh", "incremental"]
|
|
---
|
|
|
|
A materialized view is a table whose contents are defined by a query over
|
|
another table. The query is recorded in the view's schema; a refresh computes
|
|
the rows.
|
|
|
|
The result is an ordinary table. It can be queried, indexed, and searched, and
|
|
it appears in `table_names()` — nothing about reading it is special.
|
|
|
|
## Create and refresh {#create-and-refresh}
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
db = lancedb.connect(
|
|
"./.lancedb",
|
|
storage_options={"new_table_enable_stable_row_ids": "true"},
|
|
)
|
|
table = db.create_table("people", [{"name": "ada", "age": 36},
|
|
{"name": "kid", "age": 7}])
|
|
|
|
view = db.create_materialized_view(
|
|
"adults",
|
|
"people",
|
|
select=["name", ("shout", "upper(name)")],
|
|
where="age >= 18",
|
|
)
|
|
|
|
result = view.refresh()
|
|
result.rows_written
|
|
# 1
|
|
```
|
|
|
|
`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
|
|
Omitting it selects every source column, expanded against the schema as it was
|
|
at creation time. `where` filters source rows; `limit` caps the view.
|
|
|
|
## The source needs stable row ids {#stable-row-ids}
|
|
|
|
<Warning>
|
|
The source table must have been created with stable row ids:
|
|
|
|
```python
|
|
db = lancedb.connect(uri, storage_options={"new_table_enable_stable_row_ids": "true"})
|
|
```
|
|
|
|
They are what keeps the view's provenance valid when the source is compacted,
|
|
and **they cannot be enabled after a table exists**. A table created without
|
|
them cannot later become the source of a view — it has to be recreated.
|
|
</Warning>
|
|
|
|
This is the one decision here that is hard to undo, so it is worth setting the
|
|
option on any table you might later want to derive from.
|
|
|
|
## Refreshes are incremental {#incremental}
|
|
|
|
`refresh()` reconciles the source's changes into the view — rows added, changed,
|
|
or removed since the last refresh — rather than recomputing from scratch. It
|
|
falls back to a full rebuild when the changes cannot be reconciled.
|
|
|
|
Two arguments override that:
|
|
|
|
- `full=True` forces a rebuild.
|
|
- `source_version=N` refreshes against that source version instead of the
|
|
latest, which is how you make a view reproduce a known point in time.
|
|
|
|
Concurrent refreshes of one view do not duplicate rows. If two plan the same
|
|
source rows, they conflict on commit and the loser raises rather than writing
|
|
them twice — so a retry loop around `refresh()` is safe.
|
|
|
|
## Local databases only {#availability}
|
|
|
|
<Warning>
|
|
**This** materialized view — the one created through `lancedb.connect(...)` — is
|
|
supported on local databases only. Calling `create_materialized_view` on a
|
|
remote `db://` connection raises `NotImplementedError`.
|
|
|
|
Enterprise has its own materialized views, created through a different client
|
|
with the same method name. If you are working against an Enterprise deployment,
|
|
that is the one you want; see the Enterprise feature engineering documentation.
|
|
The two are not the same feature and the method name is unfortunately shared, so
|
|
it is worth being sure which client you are holding.
|
|
</Warning>
|
|
|
|
Note the direction, which is the reverse of
|
|
[functions](/feature-engineering/functions): those run only against Cloud and
|
|
Enterprise and raise locally. Neither mechanism is available on both.
|
|
|
|
## Working with the view {#working-with-the-view}
|
|
|
|
`view.table()` returns the underlying table for querying, indexing, or search.
|
|
`view.definition()` returns the recorded query — the select list, predicate, and
|
|
limit — which is how you check what a view actually contains without inferring
|
|
it from the rows.
|
|
|
|
An existing view is reopened with `db.open_materialized_view(name)`.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/computed-columns">
|
|
Derive a column instead of a whole table.
|
|
</Card>
|
|
<Card title="Versioning" icon="clock-rotate-left" href="/tables/versioning">
|
|
Every refresh commits a table version.
|
|
</Card>
|
|
</CardGroup>
|