mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Jack Ye
7e8f0d4982
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.
## Current layout
Local materialized views currently maintain incremental-refresh state
alongside the materialized values:
- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.
Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.
## New layout
An MV is an ordinary Lance backing table containing only:
- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.
There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.
Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.
## New remote create workflow
```text
Connection::create_materialized_view(...)
-> builder renders the defining SELECT
-> remote Database receives a CreateMaterializedViewRequest
-> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
-> retain the SQL Query handle while the Job is running
-> consume the durable SQL-job result
-> open the created table and read mv.definition
-> return MaterializedView
```
Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.
## API changes
- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.
The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
1.7 KiB
1.7 KiB
@lancedb/lancedb • Docs
@lancedb/lancedb / MaterializedView
Class: MaterializedView
A handle on a materialized view: its table plus its definition.
Obtained from Connection#createMaterializedView or Connection#openMaterializedView. The view is a normal table -- queries, indexes and search all apply through MaterializedView#table -- whose contents are maintained by MaterializedView#refresh.
Constructors
new MaterializedView()
new MaterializedView(table): MaterializedView
Parameters
- table:
Table
Returns
Accessors
name
get name(): string
Returns
string
Methods
definition()
definition(): Promise<MaterializedViewDefinition>
The query that defines the view.
Returns
Promise<MaterializedViewDefinition>
refresh()
refresh(options?): Promise<RefreshMaterializedViewResult>
Recompute the view from its source.
Every refresh rebuilds the complete view. For local views,
sourceVersion refreshes to that source version instead of the latest;
remote SQL refreshes do not support source-version pinning.
Parameters
-
options?
-
options.sourceVersion?:
number
Returns
Promise<RefreshMaterializedViewResult>
table()
table(): Table
The view, as the table it is.