diff --git a/docs/web/architecture.mdx b/docs/web/architecture.mdx index 28fc9e1eb..4ee1bc0e7 100644 --- a/docs/web/architecture.mdx +++ b/docs/web/architecture.mdx @@ -128,8 +128,9 @@ Because the storage layer is identical, moving is a connection-string change plus a data migration, not a rewrite. -The one thing to decide early is **stable row ids**, which cannot be enabled -after a table exists and which a deployment may require on a +The one thing to decide early is **stable row ids**: the SDK can give them to a +table only when it creates one on a local path or an object store, and a +deployment may require them on a [materialized view](/management/materialized-view) source. See [Limits](/basics/limits#schema). diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx index 4303dc08e..f135de8d4 100644 --- a/docs/web/basics/limits.mdx +++ b/docs/web/basics/limits.mdx @@ -55,10 +55,13 @@ See [Jobs](/basics/jobs). | `IVF_RQ` vector dimensions | must be divisible by 8 | | FTS `max_token_length` | 40 by default; longer tokens are dropped | -Stable row ids **cannot be enabled after a table exists** -- the one schema -decision here that is genuinely irreversible. A deployment may require them on a -[materialized view](/management/materialized-view) source; a local view does -not. +Through the SDK, a table gets **stable row ids** only at creation, from the +`new_table_enable_stable_row_ids` storage option on a local path or an object +store; no SDK call adds them later, so decide early. A local +[materialized view](/feature-engineering/materialized-views#stable-row-ids) does +not need them. On a deployment, whether a view's source needs them and how a +table gets them depend on the deployment; see +[Against a deployment](/feature-engineering/materialized-views#availability). See [Data types](/sql/data-types). @@ -73,7 +76,7 @@ See [Data types](/sql/data-types). | `TABLESAMPLE BUCKET` | not supported | | `TABLESAMPLE` on a WAL-backed table | not supported | | `CREATE INDEX` | one column, server-chosen name | -| `REFRESH MATERIALIZED VIEW ... CASCADE` | parsed, not yet implemented | +| `REFRESH MATERIALIZED VIEW ... CASCADE` | depends on the deployment's version; see [REFRESH](/sql/dml/refresh#refresh-materialized-view) | A sampling percentage over 100 is an error rather than a full scan, because a query asking for 150 percent has a mistake in it and reading it as 100 would diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx index 74bf81356..7b00a6ce4 100644 --- a/docs/web/feature-engineering/materialized-views.mdx +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -49,14 +49,17 @@ matching row, two in this example. Omitting it selects every source column, expanded against the schema as it was at creation time. `where` filters source rows; `limit` caps the view. -## Any local table can be the source {#stable-row-ids} +## Local sources no longer need stable row ids {#stable-row-ids} -Locally, the source needs no special storage option. A local refresh does not -track which source rows the view came from, so stable row ids -(`new_table_enable_stable_row_ids`) are not required: the example above creates -its source with the defaults. Earlier development builds and the published -0.39.0 required them, and a deployment may still require them; see -[Against a deployment](#availability). +A local refresh does not track which source rows the view came from, so a local +view's source does not need stable row ids (`new_table_enable_stable_row_ids`): +the example above creates its source with the defaults. Earlier development +builds and the published 0.39.0 required them, and a deployment may still +require them; see [Against a deployment](#availability). + +The source is still checked, when the view is created and on every refresh: one +with an LSM write spec, or with retained write-ahead-log rows not yet compacted, +is refused. ## Every local refresh rebuilds the view {#incremental} @@ -85,8 +88,15 @@ there. How a deployment refreshes a view -- rebuilding the whole view, as above, or reconciling what changed in the source -- depends on its LanceDB Enterprise version and configuration, and so do the options [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view) -accepts and whether the source table needs stable row ids. Where a deployment -requires them, create the source with `new_table_enable_stable_row_ids`. +accepts and whether the source table needs stable row ids. + +Stable row ids are set up on the deployment, not by the SDK: a `db://` +connection does not send `new_table_enable_stable_row_ids`, and no SDK call adds +them to an existing table. Depending on its version and configuration, a +deployment may give new tables stable row ids, may refuse a view created through +SQL -- as the SDK creates one -- over a source without them, or may convert an +existing source when a view is created through its REST API. Check with the +deployment's administrator what it requires before relying on a source. `SHOW MATERIALIZED VIEWS` lists them, which is cheaper than `list_materialized_views` -- that method finds views by reading every table's diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index fbb9a8ee1..1f77e5b10 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -106,10 +106,11 @@ const db = await lancedb.connect("s3://bucket/path", { -`new_table_enable_stable_row_ids` is worth setting deliberately: stable row ids -cannot be enabled after a table exists, and a deployment may require them on a -[materialized view](/feature-engineering/materialized-views#availability)'s -source. +`new_table_enable_stable_row_ids` is worth setting deliberately: it gives stable +row ids to the tables this connection creates, and no SDK call adds them to an +existing table. A `db://` connection does not send it; a deployment decides +stable row ids itself (see +[Against a deployment](/feature-engineering/materialized-views#availability)). See [Storage](/deployment/storage) for the full set, including per-cloud credentials. diff --git a/docs/web/management/materialized-view.mdx b/docs/web/management/materialized-view.mdx index 69828144a..cbdb703b1 100644 --- a/docs/web/management/materialized-view.mdx +++ b/docs/web/management/materialized-view.mdx @@ -40,10 +40,10 @@ const view = await db.createMaterializedView("recent_docs", "documents", { ``` -`source` names the table it derives from. Locally any table can be one, with no -special storage option; a deployment may require it to have stable row ids (see -[Against a deployment](#deployment)). `select`, `where` and `limit` record the -query -- there is no separate definition step. +`source` names the table it derives from. A local source does not need stable +row ids or any other storage option; a deployment may require stable row ids +(see [Against a deployment](#deployment)). `select`, `where` and `limit` record +the query -- there is no separate definition step. ## Opening, listing and dropping {#manage} diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index 89a50c74b..5cf72a468 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -156,9 +156,9 @@ frame, or Pydantic models. [Ingesting data](/ingestion) covers each, plus `exist_ok` and `mode="overwrite"` for re-runnable scripts. -An explicit schema is where nullability is decided, and where you opt into -**stable row ids** -- which cannot be enabled later, and which a deployment may -require on a [materialized view](/management/materialized-view) source. See +An explicit schema is where nullability is decided. Creation is also the only +point at which the SDK can give a table **stable row ids**, which a deployment +may require on a [materialized view](/management/materialized-view) source. See [Limits](/basics/limits#schema).