refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)

## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-23 04:02:27 +00:00 · 2026-10-16 13:33:57 -05:00
parent 0968c21f4c
commit fe34eb3549
73 changed files with 1853 additions and 1856 deletions
@@ -228,17 +228,17 @@ version of the table.

 ***

-### checkpointLsm()
+### checkpointMemWal()

 ```ts
-abstract checkpointLsm(): Promise<void>
+abstract checkpointMemWal(): Promise<void>
 ```

-Converge this table's LSM write path into its base table.
+Converge this table's MemWAL write path into its base table.

-Seals once, then triggers compaction and polls until the L0 that existed
+Freezes once, then triggers compaction and polls until the SSTables that existed
 at the start is gone. The target set is fixed at the start, so
-generations created *during* the checkpoint are ignored — that is what
+SSTables created *during* the checkpoint are ignored — that is what
 lets it terminate under write load, and what makes it best-effort: it
 converges the fresh tier as of some instant. Idempotent, abandonable at
 any point, and safe to run on a cadence.
@@ -254,9 +254,9 @@ is merging. The caller owns the deadline.
 #### Example

 ```ts
-const before = await table.getLsmStats();
-await table.checkpointLsm();
-const after = await table.getLsmStats();
+const before = await table.getMemWalStats();
+await table.checkpointMemWal();
+const after = await table.getMemWalStats();
 ```

 ***
@@ -279,15 +279,15 @@ Any attempt to use the table after it is closed will result in an error.

 ***

-### closeLsmWriters()
+### closeMemWalWriters()

 ```ts
-abstract closeLsmWriters(): Promise<void>
+abstract closeMemWalWriters(): Promise<void>
 ```

 Drain and close any cached MemWAL shard writers held for this table.

-When an [LsmWriteSpec](../interfaces/LsmWriteSpec.md) is installed, `mergeInsert` opens MemWAL
+When an [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) is installed, `mergeInsert` opens MemWAL
 shard writers and caches them for reuse across calls. This closes them,
 flushing pending data; writers reopen lazily on the next `mergeInsert`.
 It is a no-op when no writers are cached.
@@ -298,17 +298,17 @@ It is a no-op when no writers are cached.

 ***

-### compactLsm()
+### compactMemWal()

 ```ts
-abstract compactLsm(): Promise<void>
+abstract compactMemWal(): Promise<void>
 ```

-Trigger a background L0 → base compaction pass per bucket.
+Trigger a background SSTable compaction pass per table shard.

 Returns once the passes are *dispatched*, not once they finish — watch
-[Table#getLsmStats](Table.md#getlsmstats) for progress, or use
-[Table#checkpointLsm](Table.md#checkpointlsm) to wait for convergence.
+[Table#getMemWalStats](Table.md#getmemwalstats) for progress, or use
+[Table#checkpointMemWal](Table.md#checkpointmemwal) to wait for convergence.

 #### Returns

@@ -449,6 +449,23 @@ containing the new version number of the table

 ***

+### disableMemWal()
+
+```ts
+abstract disableMemWal(): Promise<void>
+```
+
+Remove the [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) from this table, reverting to the standard
+`mergeInsert` write path.
+
+Errors if no spec is currently set.
+
+#### Returns
+
+`Promise`&lt;`void`&gt;
+
+***
+
 ### display()

 ```ts
@@ -514,6 +531,54 @@ Drop an index from the table.

 ***

+### enableMemWal()
+
+```ts
+abstract enableMemWal(spec): Promise<void>
+```
+
+Install an [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) on this table, selecting Lance's MemWAL
+LSM-style write path for future `mergeInsert` calls.
+
+`MemWalWriteSpec` chooses one of three sharding strategies via `specType`:
+
+- `"bucket"` — hash-bucket writes by the single-column unenforced primary
+  key (`column` and `numBuckets` required).
+- `"identity"` — shard by the raw value of a scalar `column`.
+- `"unsharded"` — route every write to a single shard.
+
+All variants require the table to have an unenforced primary key
+([Table#setUnenforcedPrimaryKey](Table.md#setunenforcedprimarykey)); bucket sharding additionally
+requires it to be the single column being bucketed.
+
+Omitting `maintainedIndexes` maintains every index the table has,
+including ones created later, and skips a kind the MemWAL cannot maintain.
+Naming them pins an exact set, and a still-building index is rejected
+rather than quietly omitted.
+
+#### Parameters
+
+* **spec**: [`MemWalWriteSpec`](../interfaces/MemWalWriteSpec.md)
+    The sharding spec to install.
+
+#### Returns
+
+`Promise`&lt;`void`&gt;
+
+#### Example
+
+```ts
+await table.setUnenforcedPrimaryKey("id");
+await table.enableMemWal({
+  specType: "bucket",
+  column: "id",
+  numBuckets: 16,
+  maintainedIndexes: ["id_idx"],
+});
+```
+
+***
+
 ### fetchBlobFiles()

 ```ts
@@ -582,15 +647,15 @@ memory; use [Table.fetchBlobFiles](Table.md#fetchblobfiles) for large values.

 ***

-### flushLsm()
+### flushMemWal()

 ```ts
-abstract flushLsm(): Promise<void>
+abstract flushMemWal(): Promise<void>
 ```

-Seal every bucket's active memtable into a new L0 generation.
+Freeze every table shard's active memtable into a new SSTable.

-Returns once the seal is committed. Sealing an empty memtable is a no-op,
+Returns once the freeze is committed. Freezing an empty memtable is a no-op,
 so this is safe to call repeatedly.

 #### Returns
@@ -634,48 +699,48 @@ const { records, truncated } = await table.functionErrors({ column: "embedding"

 ***

-### getLsmStats()
+### getMemWalSpec()

 ```ts
-abstract getLsmStats(includeGenerationRows?): Promise<undefined | LsmStats>
+abstract getMemWalSpec(): Promise<undefined | MemWalWriteSpec>
 ```

-Read live per-bucket LSM state.
+Read the [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) currently installed on this table.

-Answers "how far behind is my fresh tier", "which bucket is hot", and
-"why is my fresh-tier vector search brute-force". Mutates no table state.
-
-Resolves to `undefined` only when the LSM write path is not enabled.
-
-#### Parameters
-
-* **includeGenerationRows?**: `boolean`
-    Also count rows per L0 generation.
-    Off by default because each count opens an uncached Lance dataset.
-
-#### Returns
-
-`Promise`&lt;`undefined` \| [`LsmStats`](../interfaces/LsmStats.md)&gt;
-
-***
-
-### getLsmWriteSpec()
-
-```ts
-abstract getLsmWriteSpec(): Promise<undefined | LsmWriteSpec>
-```
-
-Read the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) currently installed on this table.
-
-Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
-spec has been set, or it was removed with [Table#unsetLsmWriteSpec](Table.md#unsetlsmwritespec)).
+Resolves to `undefined` when the MemWAL write path is not enabled (no
+spec has been set, or it was removed with [Table#disableMemWal](Table.md#disablememwal)).
 The spec is the one installed, including its maintained-index selection:
 an absent `maintainedIndexes` for every index the table has, an empty
 array for none.

 #### Returns

-`Promise`&lt;`undefined` \| [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)&gt;
+`Promise`&lt;`undefined` \| [`MemWalWriteSpec`](../interfaces/MemWalWriteSpec.md)&gt;
+
+***
+
+### getMemWalStats()
+
+```ts
+abstract getMemWalStats(includeSstableRows?): Promise<undefined | MemWalStats>
+```
+
+Read live per-table-shard MemWAL state.
+
+Answers "how far behind is my fresh tier", "which table shard is hot", and
+"why is my fresh-tier vector search brute-force". Mutates no table state.
+
+Resolves to `undefined` only when the MemWAL write path is not enabled.
+
+#### Parameters
+
+* **includeSstableRows?**: `boolean`
+    Also count rows per SSTable.
+    Off by default because each count opens an uncached Lance dataset.
+
+#### Returns
+
+`Promise`&lt;`undefined` \| [`MemWalStats`](../interfaces/MemWalStats.md)&gt;

 ***

@@ -1093,54 +1158,6 @@ of the given query

 ***

-### setLsmWriteSpec()
-
-```ts
-abstract setLsmWriteSpec(spec): Promise<void>
-```
-
-Install an [LsmWriteSpec](../interfaces/LsmWriteSpec.md) on this table, selecting Lance's MemWAL
-LSM-style write path for future `mergeInsert` calls.
-
-`LsmWriteSpec` chooses one of three sharding strategies via `specType`:
-
- `"bucket"` — hash-bucket writes by the single-column unenforced primary
-  key (`column` and `numBuckets` required).
- `"identity"` — shard by the raw value of a scalar `column`.
- `"unsharded"` — route every write to a single shard.
-
-All variants require the table to have an unenforced primary key
-([Table#setUnenforcedPrimaryKey](Table.md#setunenforcedprimarykey)); bucket sharding additionally
-requires it to be the single column being bucketed.
-
-Omitting `maintainedIndexes` maintains every index the table has,
-including ones created later, and skips a kind the MemWAL cannot maintain.
-Naming them pins an exact set, and a still-building index is rejected
-rather than quietly omitted.
-
-#### Parameters
-
-* **spec**: [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)
-    The sharding spec to install.
-
-#### Returns
-
-`Promise`&lt;`void`&gt;
-
-#### Example
-
-```ts
-await table.setUnenforcedPrimaryKey("id");
-await table.setLsmWriteSpec({
-  specType: "bucket",
-  column: "id",
-  numBuckets: 16,
-  maintainedIndexes: ["id_idx"],
-});
-```
-
-***
-
 ### setUnenforcedPrimaryKey()

 ```ts
@@ -1295,23 +1312,6 @@ same tokenizer model files must also exist locally.

 ***

-### unsetLsmWriteSpec()
-
-```ts
-abstract unsetLsmWriteSpec(): Promise<void>
-```
-
-Remove the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) from this table, reverting to the standard
-`mergeInsert` write path.
-
-Errors if no spec is currently set.
-
-#### Returns
-
-`Promise`&lt;`void`&gt;
-
-***
-
 ### update()

 #### update(opts)