diff --git a/docs/web/architecture.mdx b/docs/web/architecture.mdx index 1a3d85d65..28fc9e1eb 100644 --- a/docs/web/architecture.mdx +++ b/docs/web/architecture.mdx @@ -129,7 +129,9 @@ plus a data migration, not a rewrite. The one thing to decide early is **stable row ids**, which cannot be enabled -after a table exists. See [Limits](/basics/limits#schema). +after a table exists and which a deployment may require on a +[materialized view](/management/materialized-view) source. See +[Limits](/basics/limits#schema). ## Where to go next {#next} diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx index d7afbe199..4303dc08e 100644 --- a/docs/web/basics/limits.mdx +++ b/docs/web/basics/limits.mdx @@ -56,7 +56,9 @@ See [Jobs](/basics/jobs). | FTS `max_token_length` | 40 by default; longer tokens are dropped | Stable row ids **cannot be enabled after a table exists** -- the one schema -decision here that is genuinely irreversible. +decision here that is genuinely irreversible. A deployment may require them on a +[materialized view](/management/materialized-view) source; a local view does +not. See [Data types](/sql/data-types). diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index 5a9cb45a7..4b8200d40 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -54,8 +54,8 @@ idempotent and cheap -- but it will surprise you if you expect a spreadsheet. expression, filled by [a refresh](/feature-engineering/refresh-column). **Materialized views** derive a whole table -- a filtered, projected, or -aggregated view of a source -- and compute it again on each refresh. The result -is an ordinary table: queryable, indexable, searchable. +aggregated view of a source -- and bring it up to date on each refresh. The +result is an ordinary table: queryable, indexable, searchable. **Functions** run Python where the data is, for work SQL cannot express: embedding a column, calling a model, decoding an image. Registered once, diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx index b0e7e142b..74bf81356 100644 --- a/docs/web/feature-engineering/materialized-views.mdx +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -1,7 +1,7 @@ --- title: "Refresh a materialized view" sidebarTitle: "Materialized View Refresh" -description: "A lakehouse object: a table defined by a query over another table, recomputed by each refresh." +description: "A lakehouse object: a table defined by a query over another table, brought up to date by each refresh." icon: table keywords: ["materialized view", "create_materialized_view", "refresh", "full refresh"] --- @@ -13,7 +13,7 @@ import { A materialized view is a lakehouse object: a table whose contents are defined by a query over another table. The query is recorded in the view's schema; -creating the view computes its rows, and a refresh computes them again. +creating the view computes its rows, and a refresh brings them up to date. The result is an ordinary table. It can be queried, indexed, and searched, and it appears in `table_names()` — nothing about reading it is special. @@ -49,25 +49,26 @@ matching row, two in this example. Omitting it selects every source column, expanded against the schema as it was at creation time. `where` filters source rows; `limit` caps the view. -## Any table can be the source {#stable-row-ids} +## Any local table can be the source {#stable-row-ids} -The source needs no special storage option. A refresh does not track which -source rows the view came from, so stable row ids +Locally, the source needs no special storage option. A local refresh does not +track which source rows the view came from, so stable row ids (`new_table_enable_stable_row_ids`) are not required: the example above creates -its source with the defaults. Earlier development builds, and the published -0.39.0, required them. +its source with the defaults. Earlier development builds and the published +0.39.0 required them, and a deployment may still require them; see +[Against a deployment](#availability). -## Every refresh rebuilds the view {#incremental} +## Every local refresh rebuilds the view {#incremental} -`refresh()` runs the view's query against the source and replaces everything -the view holds with the result, in one commit. It does not reconcile changes: -`rows_written` is the size of the whole result — two after adding Bea, not -one — and a refresh with nothing new to read still rewrites the view and +Locally, `refresh()` runs the view's query against the source and replaces +everything the view holds with the result, in one commit. It does not reconcile +changes: `rows_written` is the size of the whole result — two after adding Bea, +not one — and a refresh with nothing new to read still rewrites the view and commits a new table version. `source_version=N` refreshes against that source version instead of the latest, which is how a view reproduces a known point in time. There is no `full` -argument; every refresh is a full one. +argument; every local refresh is a full one. ## Against a deployment {#availability} @@ -84,7 +85,8 @@ there. How a deployment refreshes a view -- rebuilding the whole view, as above, or reconciling what changed in the source -- depends on its LanceDB Enterprise version and configuration, and so do the options [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view) -accepts. +accepts and whether the source table needs stable row ids. Where a deployment +requires them, create the source with `new_table_enable_stable_row_ids`. `SHOW MATERIALIZED VIEWS` lists them, which is cheaper than `list_materialized_views` -- that method finds views by reading every table's diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index 83a249f91..fbb9a8ee1 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -107,7 +107,9 @@ const db = await lancedb.connect("s3://bucket/path", { `new_table_enable_stable_row_ids` is worth setting deliberately: stable row ids -cannot be enabled after a table exists. +cannot be enabled after a table exists, and a deployment may require them on a +[materialized view](/feature-engineering/materialized-views#availability)'s +source. See [Storage](/deployment/storage) for the full set, including per-cloud credentials. diff --git a/docs/web/management/materialized-view.mdx b/docs/web/management/materialized-view.mdx index bc7aa92c5..69828144a 100644 --- a/docs/web/management/materialized-view.mdx +++ b/docs/web/management/materialized-view.mdx @@ -8,7 +8,7 @@ keywords: ["materialized view", "create_materialized_view", "catalog", "ddl"] A materialized view is a table whose contents are defined by a query over another table. The query is recorded in the view's schema, and a refresh -computes its stored rows again from the source. +brings its stored rows up to date with the source. The result is an ordinary table -- queryable, indexable, searchable, and present in `table_names()`. Nothing about reading it is special. @@ -40,9 +40,10 @@ const view = await db.createMaterializedView("recent_docs", "documents", { ``` -`source` names the table it derives from; any table can be one, with no special -storage option. `select`, `where` and `limit` record the query -- there is no -separate definition step. +`source` names the table it derives from. Locally any table can be one, with no +special storage option; a deployment may require it to have stable row ids (see +[Against a deployment](#deployment)). `select`, `where` and `limit` record the +query -- there is no separate definition step. ## Opening, listing and dropping {#manage} @@ -78,14 +79,15 @@ rows. Both are properties; on an async view, the definition is read with `db://` connection as well as a local one. There the SDK creates and refreshes the view through SQL, and the work runs on the cluster rather than in your process. Materialized views are off by default on a deployment and have to be -enabled there; see +enabled there, and a deployment may require a view's source to have stable row +ids; see [Against a deployment](/feature-engineering/materialized-views#availability). ## What to read next {#next} - What a refresh recomputes, and what it writes. + How a refresh works, and what it writes. The non-materialized counterpart, defined in SQL. diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index f85bfe9b1..89a50c74b 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -157,7 +157,8 @@ frame, or Pydantic models. [Ingesting data](/ingestion) covers each, plus An explicit schema is where nullability is decided, and where you opt into -**stable row ids** -- which cannot be enabled later. See +**stable row ids** -- which cannot be enabled later, and which a deployment may +require on a [materialized view](/management/materialized-view) source. See [Limits](/basics/limits#schema). diff --git a/docs/web/sql/ddl/materialized-view.mdx b/docs/web/sql/ddl/materialized-view.mdx index 9c68af2f2..51b5ef55b 100644 --- a/docs/web/sql/ddl/materialized-view.mdx +++ b/docs/web/sql/ddl/materialized-view.mdx @@ -13,8 +13,9 @@ A materialized view stores a query's result as rows, kept current by a refresh Materialized views are off by default on a deployment and have to be enabled -there. Which statements and query shapes a deployment accepts, and how it -refreshes a view, depend on its LanceDB Enterprise version and configuration. +there. Which statements and query shapes a deployment accepts, whether it needs +the source table to have stable row ids, and how it refreshes a view depend on +its LanceDB Enterprise version and configuration. ## CREATE MATERIALIZED VIEW {#create} @@ -67,6 +68,6 @@ as any table. Creating one from the SDK. - What a refresh recomputes, locally and on a deployment. + How a refresh works, locally and on a deployment. diff --git a/docs/web/tutorials/index.mdx b/docs/web/tutorials/index.mdx index bcc9345c1..6ada57c9a 100644 --- a/docs/web/tutorials/index.mdx +++ b/docs/web/tutorials/index.mdx @@ -38,5 +38,5 @@ are runnable in Colab; the rest are written up here. | [Multi-vector search](/search/vector-search) | Several embeddings per document. | | [Reranking](/reranking) | Reordering results to improve the top-k. | | [Feature engineering](/feature-engineering) | Deriving columns and tables from data already stored. | -| [Materialized views](/management/materialized-view) | A table defined by a query, recomputed by each refresh. | +| [Materialized views](/management/materialized-view) | A table defined by a query, brought up to date by each refresh. | | [Training](/training/data-loading) | Reading a table into a data loader. |