mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
187 lines
6.2 KiB
Plaintext
187 lines
6.2 KiB
Plaintext
---
|
|
title: "Create a materialized view"
|
|
sidebarTitle: "Create Materialized View"
|
|
description: "Store the result of a query as a table, ready to query, index and search."
|
|
icon: table
|
|
keywords: ["materialized view", "create_materialized_view"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A materialized view stores the result of a query as a table. Use it to keep a
|
|
filtered or transformed dataset that you can query, index and search
|
|
repeatedly, instead of recomputing the query every time you read it.
|
|
|
|
<Note>
|
|
In SQL this is [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view#create).
|
|
</Note>
|
|
|
|
## Create a view {#create}
|
|
|
|
A materialized view is defined by a SQL query, so it is created with one.
|
|
From Python, run the statement on an [Enterprise connection](/basics/catalog#enterprise)
|
|
with [`execute_query`](/basics/sql#sql-from-python); the SQL is the same
|
|
either way:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
db.execute_query("""
|
|
CREATE MATERIALIZED VIEW frames_captioned AS
|
|
SELECT frame_id, video_id, ts, caption
|
|
FROM frames
|
|
WHERE selected = true
|
|
""").read_all()
|
|
view = db.open_materialized_view("frames_captioned")
|
|
```
|
|
|
|
```sql SQL icon="database"
|
|
CREATE MATERIALIZED VIEW frames_captioned AS
|
|
SELECT frame_id, video_id, ts, caption
|
|
FROM frames
|
|
WHERE selected = true
|
|
```
|
|
</CodeGroup>
|
|
|
|
`execute_query` returns once the result stream is readable, not once the
|
|
statement has finished, so `.read_all()` waits out the full creation before
|
|
the next line opens it.
|
|
|
|
The query defines the view: here, the `frames` table's `frame_id`,
|
|
`video_id`, `ts` and `caption` columns, filtered to the frames selected for
|
|
captioning. There is no separate, more limited form for defining one --
|
|
joins and [Functions](#functions) in the select list work the same way any
|
|
other `SELECT` does, within the query shapes your deployment's LanceDB
|
|
Enterprise version and configuration accept; see the
|
|
[SQL reference](/sql/ddl/materialized-view#create).
|
|
|
|
Creating the view computes its rows, so it is ready to read immediately.
|
|
[`db.open_materialized_view`](/management/materialized-view#manage) gets a
|
|
handle back for reading, refreshing or inspecting it from Python; the view
|
|
itself is also an ordinary table, queryable by name like any other.
|
|
|
|
Your administrator manages stable row IDs and other source-table
|
|
requirements for materialized views on the server; confirm support for your
|
|
source table's shape before creating a view over it.
|
|
|
|
## Selecting a Function's output {#functions}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
A `SELECT` expression is SQL text, so it can call a registered
|
|
[Function](/curation/create-function) the same way a computed column's
|
|
expression can. Continuing the
|
|
[`caption` Function](/curation/create-function#chain) registered in Create a
|
|
Function, against a `frames` table that already has a `jpeg` column:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
db.execute_query("""
|
|
CREATE MATERIALIZED VIEW frames_captioned AS
|
|
SELECT frame_id, jpeg, caption(jpeg) AS caption
|
|
FROM frames
|
|
WHERE selected = true
|
|
""").read_all()
|
|
```
|
|
|
|
```sql SQL icon="database"
|
|
CREATE MATERIALIZED VIEW frames_captioned AS
|
|
SELECT frame_id, jpeg, caption(jpeg) AS caption
|
|
FROM frames
|
|
WHERE selected = true
|
|
```
|
|
</CodeGroup>
|
|
|
|
A query that joins tables works the same way, with no separate syntax for
|
|
it. Continuing the join and chained Functions from
|
|
[Create a Function](/curation/create-function#chain), producing `jpeg` from
|
|
a separate `videos` table and captioning it in the same statement:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
db.execute_query("""
|
|
CREATE MATERIALIZED VIEW frames_captioned
|
|
WITH (auto_refresh_enabled = false, auto_refresh_interval_seconds = 5)
|
|
AS
|
|
SELECT
|
|
f.frame_id,
|
|
extract_frame(v.video, f.ts) AS jpeg,
|
|
caption(extract_frame(v.video, f.ts)) AS caption
|
|
FROM videos v
|
|
JOIN frames f ON v.id = f.video_id
|
|
WHERE f.selected = true
|
|
""").read_all()
|
|
```
|
|
|
|
```sql SQL icon="database"
|
|
CREATE MATERIALIZED VIEW frames_captioned
|
|
WITH (auto_refresh_enabled = false, auto_refresh_interval_seconds = 5)
|
|
AS
|
|
SELECT
|
|
f.frame_id,
|
|
extract_frame(v.video, f.ts) AS jpeg,
|
|
caption(extract_frame(v.video, f.ts)) AS caption
|
|
FROM videos v
|
|
JOIN frames f ON v.id = f.video_id
|
|
WHERE f.selected = true
|
|
```
|
|
</CodeGroup>
|
|
|
|
See [Joins](/curation/join) for the join itself, and
|
|
[Auto Refresh](/enterprise/auto-refresh) for what `WITH` sets here.
|
|
|
|
## Automatic refresh at creation {#auto-refresh}
|
|
|
|
Set `auto_refresh_enabled` and `auto_refresh_interval_seconds` in the `WITH`
|
|
clause to schedule refreshes instead of calling
|
|
[`refresh`](/feature-engineering/refresh-materialized-view) yourself:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
db.execute_query("""
|
|
CREATE MATERIALIZED VIEW frames_captioned
|
|
WITH (auto_refresh_enabled = true, auto_refresh_interval_seconds = 300)
|
|
AS
|
|
SELECT frame_id, video_id, ts, caption
|
|
FROM frames
|
|
WHERE selected = true
|
|
""").read_all()
|
|
```
|
|
|
|
```sql SQL icon="database"
|
|
CREATE MATERIALIZED VIEW frames_captioned
|
|
WITH (auto_refresh_enabled = true, auto_refresh_interval_seconds = 300)
|
|
AS
|
|
SELECT frame_id, video_id, ts, caption
|
|
FROM frames
|
|
WHERE selected = true
|
|
```
|
|
</CodeGroup>
|
|
|
|
Omit `auto_refresh_enabled` to let the server decide whether to enable it,
|
|
and omit `auto_refresh_interval_seconds` to let the server decide the
|
|
pacing. See [Auto Refresh](/enterprise/auto-refresh) for how scheduling
|
|
behaves once the view exists.
|
|
|
|
## Working with the view {#working-with-the-view}
|
|
|
|
`view.table` is the underlying table, for querying, indexing, or search.
|
|
`view.definition` has a single field, `query` -- the recorded query, in the
|
|
canonical spelling the server stores. Use it to inspect the query without
|
|
inferring it from the rows. Both are properties; on an async view, the
|
|
definition is read with `await view.definition()`.
|
|
|
|
Creation, refresh and drop also have async, job-returning SDK methods; see
|
|
[Materialized view](/management/materialized-view#manage).
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/refresh-materialized-view">
|
|
Bring the view up to date with its source.
|
|
</Card>
|
|
<Card title="Versioning" icon="clock-rotate-left" href="/management/table#versioning">
|
|
Inspect the table history and read a particular version.
|
|
</Card>
|
|
</CardGroup>
|