mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-15 08:12:26 +00:00
docs: keep the stable-row-id caveat for deployments
The earlier change dropped every statement that a materialized view's
source needs stable row ids, because main no longer requires them. That
holds for the local SDK only. Sophon main creates a view through SQL by
default with the creation checks of its pinned lancedb (00f5dd6f), which
predates #4373 and still rejects a source without stable row ids.
The refresh and management pages now say a local view needs no special
source and a deployment may still require stable row ids, and the
architecture, limits, table and database pages note that requirement.
The summaries say a refresh brings a view up to date instead of saying
it recomputes it, since a deployment may reconcile changes rather than
rebuild.
This commit is contained in:
@@ -129,7 +129,9 @@ plus a data migration, not a rewrite.
|
||||
|
||||
<Note>
|
||||
The one thing to decide early is **stable row ids**, which cannot be enabled
|
||||
after a table exists. See [Limits](/basics/limits#schema).
|
||||
after a table exists and which a deployment may require on a
|
||||
[materialized view](/management/materialized-view) source. See
|
||||
[Limits](/basics/limits#schema).
|
||||
</Note>
|
||||
|
||||
## Where to go next {#next}
|
||||
|
||||
@@ -56,7 +56,9 @@ See [Jobs](/basics/jobs).
|
||||
| FTS `max_token_length` | 40 by default; longer tokens are dropped |
|
||||
|
||||
Stable row ids **cannot be enabled after a table exists** -- the one schema
|
||||
decision here that is genuinely irreversible.
|
||||
decision here that is genuinely irreversible. A deployment may require them on a
|
||||
[materialized view](/management/materialized-view) source; a local view does
|
||||
not.
|
||||
|
||||
See [Data types](/sql/data-types).
|
||||
|
||||
|
||||
@@ -54,8 +54,8 @@ idempotent and cheap -- but it will surprise you if you expect a spreadsheet.
|
||||
expression, filled by [a refresh](/feature-engineering/refresh-column).
|
||||
|
||||
**Materialized views** derive a whole table -- a filtered, projected, or
|
||||
aggregated view of a source -- and compute it again on each refresh. The result
|
||||
is an ordinary table: queryable, indexable, searchable.
|
||||
aggregated view of a source -- and bring it up to date on each refresh. The
|
||||
result is an ordinary table: queryable, indexable, searchable.
|
||||
|
||||
**Functions** run Python where the data is, for work SQL cannot express:
|
||||
embedding a column, calling a model, decoding an image. Registered once,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: "Refresh a materialized view"
|
||||
sidebarTitle: "Materialized View Refresh"
|
||||
description: "A lakehouse object: a table defined by a query over another table, recomputed by each refresh."
|
||||
description: "A lakehouse object: a table defined by a query over another table, brought up to date by each refresh."
|
||||
icon: table
|
||||
keywords: ["materialized view", "create_materialized_view", "refresh", "full refresh"]
|
||||
---
|
||||
@@ -13,7 +13,7 @@ import {
|
||||
|
||||
A materialized view is a lakehouse object: a table whose contents are defined by
|
||||
a query over another table. The query is recorded in the view's schema;
|
||||
creating the view computes its rows, and a refresh computes them again.
|
||||
creating the view computes its rows, and a refresh brings them up to date.
|
||||
|
||||
The result is an ordinary table. It can be queried, indexed, and searched, and
|
||||
it appears in `table_names()` — nothing about reading it is special.
|
||||
@@ -49,25 +49,26 @@ matching row, two in this example.
|
||||
Omitting it selects every source column, expanded against the schema as it was
|
||||
at creation time. `where` filters source rows; `limit` caps the view.
|
||||
|
||||
## Any table can be the source {#stable-row-ids}
|
||||
## Any local table can be the source {#stable-row-ids}
|
||||
|
||||
The source needs no special storage option. A refresh does not track which
|
||||
source rows the view came from, so stable row ids
|
||||
Locally, the source needs no special storage option. A local refresh does not
|
||||
track which source rows the view came from, so stable row ids
|
||||
(`new_table_enable_stable_row_ids`) are not required: the example above creates
|
||||
its source with the defaults. Earlier development builds, and the published
|
||||
0.39.0, required them.
|
||||
its source with the defaults. Earlier development builds and the published
|
||||
0.39.0 required them, and a deployment may still require them; see
|
||||
[Against a deployment](#availability).
|
||||
|
||||
## Every refresh rebuilds the view {#incremental}
|
||||
## Every local refresh rebuilds the view {#incremental}
|
||||
|
||||
`refresh()` runs the view's query against the source and replaces everything
|
||||
the view holds with the result, in one commit. It does not reconcile changes:
|
||||
`rows_written` is the size of the whole result — two after adding Bea, not
|
||||
one — and a refresh with nothing new to read still rewrites the view and
|
||||
Locally, `refresh()` runs the view's query against the source and replaces
|
||||
everything the view holds with the result, in one commit. It does not reconcile
|
||||
changes: `rows_written` is the size of the whole result — two after adding Bea,
|
||||
not one — and a refresh with nothing new to read still rewrites the view and
|
||||
commits a new table version.
|
||||
|
||||
`source_version=N` refreshes against that source version instead of the latest,
|
||||
which is how a view reproduces a known point in time. There is no `full`
|
||||
argument; every refresh is a full one.
|
||||
argument; every local refresh is a full one.
|
||||
|
||||
## Against a deployment {#availability}
|
||||
|
||||
@@ -84,7 +85,8 @@ there. How a deployment refreshes a view -- rebuilding the whole view, as above,
|
||||
or reconciling what changed in the source -- depends on its LanceDB Enterprise
|
||||
version and configuration, and so do the options
|
||||
[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view)
|
||||
accepts.
|
||||
accepts and whether the source table needs stable row ids. Where a deployment
|
||||
requires them, create the source with `new_table_enable_stable_row_ids`.
|
||||
|
||||
`SHOW MATERIALIZED VIEWS` lists them, which is cheaper than
|
||||
`list_materialized_views` -- that method finds views by reading every table's
|
||||
|
||||
@@ -107,7 +107,9 @@ const db = await lancedb.connect("s3://bucket/path", {
|
||||
|
||||
<Warning>
|
||||
`new_table_enable_stable_row_ids` is worth setting deliberately: stable row ids
|
||||
cannot be enabled after a table exists.
|
||||
cannot be enabled after a table exists, and a deployment may require them on a
|
||||
[materialized view](/feature-engineering/materialized-views#availability)'s
|
||||
source.
|
||||
</Warning>
|
||||
|
||||
See [Storage](/deployment/storage) for the full set, including per-cloud credentials.
|
||||
|
||||
@@ -8,7 +8,7 @@ keywords: ["materialized view", "create_materialized_view", "catalog", "ddl"]
|
||||
|
||||
A materialized view is a table whose contents are defined by a query over
|
||||
another table. The query is recorded in the view's schema, and a refresh
|
||||
computes its stored rows again from the source.
|
||||
brings its stored rows up to date with the source.
|
||||
|
||||
The result is an ordinary table -- queryable, indexable, searchable, and present
|
||||
in `table_names()`. Nothing about reading it is special.
|
||||
@@ -40,9 +40,10 @@ const view = await db.createMaterializedView("recent_docs", "documents", {
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`source` names the table it derives from; any table can be one, with no special
|
||||
storage option. `select`, `where` and `limit` record the query -- there is no
|
||||
separate definition step.
|
||||
`source` names the table it derives from. Locally any table can be one, with no
|
||||
special storage option; a deployment may require it to have stable row ids (see
|
||||
[Against a deployment](#deployment)). `select`, `where` and `limit` record the
|
||||
query -- there is no separate definition step.
|
||||
|
||||
## Opening, listing and dropping {#manage}
|
||||
|
||||
@@ -78,14 +79,15 @@ rows. Both are properties; on an async view, the definition is read with
|
||||
`db://` connection as well as a local one. There the SDK creates and refreshes
|
||||
the view through SQL, and the work runs on the cluster rather than in your
|
||||
process. Materialized views are off by default on a deployment and have to be
|
||||
enabled there; see
|
||||
enabled there, and a deployment may require a view's source to have stable row
|
||||
ids; see
|
||||
[Against a deployment](/feature-engineering/materialized-views#availability).
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
|
||||
What a refresh recomputes, and what it writes.
|
||||
How a refresh works, and what it writes.
|
||||
</Card>
|
||||
<Card title="Views" icon="eye" href="/management/view">
|
||||
The non-materialized counterpart, defined in SQL.
|
||||
|
||||
@@ -157,7 +157,8 @@ frame, or Pydantic models. [Ingesting data](/ingestion) covers each, plus
|
||||
|
||||
<Warning>
|
||||
An explicit schema is where nullability is decided, and where you opt into
|
||||
**stable row ids** -- which cannot be enabled later. See
|
||||
**stable row ids** -- which cannot be enabled later, and which a deployment may
|
||||
require on a [materialized view](/management/materialized-view) source. See
|
||||
[Limits](/basics/limits#schema).
|
||||
</Warning>
|
||||
|
||||
|
||||
@@ -13,8 +13,9 @@ A materialized view stores a query's result as rows, kept current by a refresh
|
||||
|
||||
<Note>
|
||||
Materialized views are off by default on a deployment and have to be enabled
|
||||
there. Which statements and query shapes a deployment accepts, and how it
|
||||
refreshes a view, depend on its LanceDB Enterprise version and configuration.
|
||||
there. Which statements and query shapes a deployment accepts, whether it needs
|
||||
the source table to have stable row ids, and how it refreshes a view depend on
|
||||
its LanceDB Enterprise version and configuration.
|
||||
</Note>
|
||||
|
||||
## CREATE MATERIALIZED VIEW {#create}
|
||||
@@ -67,6 +68,6 @@ as any table.
|
||||
Creating one from the SDK.
|
||||
</Card>
|
||||
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
|
||||
What a refresh recomputes, locally and on a deployment.
|
||||
How a refresh works, locally and on a deployment.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -38,5 +38,5 @@ are runnable in Colab; the rest are written up here.
|
||||
| [Multi-vector search](/search/vector-search) | Several embeddings per document. |
|
||||
| [Reranking](/reranking) | Reordering results to improve the top-k. |
|
||||
| [Feature engineering](/feature-engineering) | Deriving columns and tables from data already stored. |
|
||||
| [Materialized views](/management/materialized-view) | A table defined by a query, recomputed by each refresh. |
|
||||
| [Materialized views](/management/materialized-view) | A table defined by a query, brought up to date by each refresh. |
|
||||
| [Training](/training/data-loading) | Reading a table into a data loader. |
|
||||
|
||||
Reference in New Issue
Block a user