Files
lancedb/docs/web/feature-engineering/materialized-views.mdx
T
Dan RammerandClaude Opus 5 fe34eb3549 refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What

Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.

### 1. MemWAL APIs drop "LSM"

The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.

| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |

TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.

Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.

### 2. Stats use SSTable and table shard

Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.

**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.

**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.

| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |

Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.

`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.

## Compatibility

Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:

- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies

## Validation

- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-16 13:33:57 -05:00

118 lines
5.0 KiB
Plaintext

---
title: "Refresh a materialized view"
sidebarTitle: "Materialized View Refresh"
description: "Create a materialized view and refresh it when the source data changes."
icon: table
keywords: ["materialized view", "create_materialized_view", "refresh", "full refresh"]
availability:
oss: available
enterprise: available
summary: >-
Both offerings support materialized views. OSS refreshes run in your process; Enterprise runs the work on remote compute. See [Enterprise refreshes](#availability) for server compatibility requirements.
---
import {
PyMaterializedViewCreate as MaterializedViewCreate,
PyMaterializedViewRefresh as MaterializedViewRefresh,
} from '/snippets/materialized_view.mdx';
A materialized view stores the result of a query as a table. Use it to keep a
filtered or transformed dataset that you can query, index and search repeatedly.
Refresh the view after its source data changes.
<Note>
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view).
</Note>
## Create and refresh {#create-and-refresh}
<CodeBlock filename="Python" language="Python" icon="python">
{MaterializedViewCreate}
</CodeBlock>
Creating the view computes its rows, so it is ready to read. Pass
`with_no_data=True` to create it empty instead. Later changes to the source
reach the view through a refresh:
<CodeBlock filename="Python" language="Python" icon="python">
{MaterializedViewRefresh}
</CodeBlock>
The example works with Python SDK 0.40.0. It creates the source with stable
row IDs, which that release requires for ordinary materialized-view sources.
`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
Omitting it selects every source column, expanded against the schema as it was
at creation time. `where` filters source rows; `limit` caps the view.
## Prepare the source table {#stable-row-ids}
In Python SDK 0.40.0, create the source with
`storage_options={"new_table_enable_stable_row_ids": "true"}`, as the example
above does. Stable row IDs let the view track source rows across updates and
compaction. Set this option when creating the table; it cannot be added later.
A source with a MemWAL write spec or retained, uncompacted write-ahead-log
rows is not accepted by the local materialized-view implementation.
## Refresh behavior {#incremental}
With Python SDK 0.40.0, a refresh applies source changes incrementally when the
query supports it, and rebuilds the view otherwise. In the example, adding Bea
writes one new view row. Refreshing again without source changes writes none.
Use `refresh(full=True)` to force a rebuild.
`rows_written` counts rows written by that refresh, not all rows in the view.
Use `view.table.count_rows()` to get the total. Pass `source_version=N` to
refresh against a particular source version instead of the latest.
<Note>
Refresh behavior has changed between SDK versions. The earlier 0.40.0b13
development build rebuilt every local view on refresh and did not accept
`full`; 0.39.0 created views empty. Use the reference for your installed SDK
when maintaining an existing pipeline.
</Note>
## Enterprise refreshes {#availability}
<Badge color="red">Enterprise</Badge>
Enterprise runs materialized-view creation and refresh on remote compute.
For SQL, see [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view) and
[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view).
Use an SDK version supported by your Enterprise deployment. Materialized-view
APIs and stored definitions have changed across releases; a local SDK example
alone does not establish compatibility with an older server. Before upgrading
an existing pipeline, confirm the supported refresh options and whether its
views need to be recreated with your administrator.
Stable row IDs and feature configuration for Enterprise sources are managed on
the server. The source-table storage option in the local example does not
configure the Enterprise deployment, and Python SDK 0.40.0's `create_table` on a
`db://` connection does not accept `storage_options`: leave it out, and confirm
with your administrator that the deployment gives new tables stable row IDs.
`list_materialized_views` requests the server's view listing. On a local
database it reads table schemas to find views.
## Working with the view {#working-with-the-view}
`view.table` is the underlying table, for querying, indexing, or search.
`view.definition` is the recorded query: the select list, predicate, and
limit. Use it to inspect the query without inferring it from the rows. Both are properties; on an async view, the definition is read
with `await view.definition()`.
An existing view is reopened with `db.open_materialized_view(name)`.
## Related guides {#next}
<CardGroup cols={2}>
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
Derive a column instead of a whole table.
</Card>
<Card title="Versioning" icon="clock-rotate-left" href="/management/table#versioning">
Inspect the table history and read a particular version.
</Card>
</CardGroup>