mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-26 13:36:13 +00:00
refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dan Rammer
co-authored by
Claude Opus 5
Claude Opus 5
parent
0968c21f4c
commit
fe34eb3549
@@ -167,10 +167,10 @@ export {
|
||||
WriteProgress,
|
||||
FtsToken,
|
||||
TokenizeTableOptions,
|
||||
LsmWriteSpec,
|
||||
LsmStats,
|
||||
BucketStats,
|
||||
GenerationStats,
|
||||
MemWalWriteSpec,
|
||||
MemWalStats,
|
||||
TableShardStats,
|
||||
SsTableStats,
|
||||
MemtableStats,
|
||||
ColumnAlteration,
|
||||
FieldMetadataUpdate,
|
||||
|
||||
@@ -90,25 +90,25 @@ export class MergeInsertBuilder {
|
||||
/**
|
||||
* Control MemWAL routing for this merge.
|
||||
*
|
||||
* By default (unset), a `mergeInsert` on a table with an LSM write spec is
|
||||
* By default (unset), a `mergeInsert` on a table with a MemWAL write spec is
|
||||
* routed through Lance's MemWAL shard writer, and a table without one uses the
|
||||
* standard path.
|
||||
*
|
||||
* @param enable - `true` forces MemWAL routing and errors if the table has no
|
||||
* LSM write spec. `false` forces the standard write path even when a spec is set.
|
||||
* MemWAL write spec. `false` forces the standard write path even when a spec is set.
|
||||
*/
|
||||
useLsm(enable: boolean): MergeInsertBuilder {
|
||||
return new MergeInsertBuilder(this.#native.useLsm(enable), this.#schema);
|
||||
useMemWal(enable: boolean): MergeInsertBuilder {
|
||||
return new MergeInsertBuilder(this.#native.useMemWal(enable), this.#schema);
|
||||
}
|
||||
/**
|
||||
* Controls how an LSM merge checks that its input targets a single shard.
|
||||
*
|
||||
* When a table has an LSM write spec, every row in a `mergeInsert` call must
|
||||
* When a table has a MemWAL write spec, every row in a `mergeInsert` call must
|
||||
* route to the same shard. When `true` (the default), every row is inspected
|
||||
* to verify this. When `false`, only the first row is inspected and the
|
||||
* shard it routes to is used for the whole input — a faster path for callers
|
||||
* that have already pre-sharded their input. Has no effect on tables without
|
||||
* an LSM write spec.
|
||||
* a MemWAL write spec.
|
||||
*
|
||||
* @param validateSingleShard - Whether to check every row routes to one shard. Defaults to `true`.
|
||||
*/
|
||||
|
||||
@@ -464,10 +464,10 @@ export class StandardQueryBase<
|
||||
* Control MemWAL read routing for this query.
|
||||
*
|
||||
* By default (unset), when the table carries a MemWAL write spec (see
|
||||
* {@link Table#setLsmWriteSpec}), reads are routed through the LSM scanner so
|
||||
* {@link Table#enableMemWal}), reads are routed through the LSM scanner so
|
||||
* they also return data written via the `mergeInsert` LSM path that has not yet
|
||||
* been compacted into the base table (the active/frozen in-memory memtables and
|
||||
* the flushed generations), deduplicated by primary key; a table without a spec
|
||||
* the SSTables), deduplicated by primary key; a table without a spec
|
||||
* reads the base table.
|
||||
*
|
||||
* @param enable - `true` forces the LSM scanner and errors if the table has no
|
||||
@@ -476,11 +476,11 @@ export class StandardQueryBase<
|
||||
*
|
||||
* Note: the LSM scanner does not support every query shape (e.g. reranking,
|
||||
* hybrid search, `orderBy`). On a MemWAL table those shapes error unless
|
||||
* `useLsm(false)` is set, because a base-only read would silently exclude
|
||||
* `useMemWal(false)` is set, because a base-only read would silently exclude
|
||||
* un-compacted MemWAL data.
|
||||
*/
|
||||
useLsm(enable: boolean): this {
|
||||
this.doCall((inner: NativeQueryType) => inner.useLsm(enable));
|
||||
useMemWal(enable: boolean): this {
|
||||
this.doCall((inner: NativeQueryType) => inner.useMemWal(enable));
|
||||
return this;
|
||||
}
|
||||
}
|
||||
@@ -871,8 +871,8 @@ export class TakeQuery extends QueryBase<NativeTakeQuery> {
|
||||
*
|
||||
* @param enable - `false` reads the base table only.
|
||||
*/
|
||||
useLsm(enable: boolean): this {
|
||||
this.doCall((inner: NativeTakeQuery) => inner.useLsm(enable));
|
||||
useMemWal(enable: boolean): this {
|
||||
this.doCall((inner: NativeTakeQuery) => inner.useMemWal(enable));
|
||||
return this;
|
||||
}
|
||||
}
|
||||
|
||||
+60
-60
@@ -35,7 +35,7 @@ import {
|
||||
FunctionErrorsOptions,
|
||||
IndexConfig,
|
||||
IndexStatistics,
|
||||
LsmStats,
|
||||
MemWalStats,
|
||||
Branches as NativeBranches,
|
||||
OptimizeStats,
|
||||
RefreshColumnResult,
|
||||
@@ -59,9 +59,9 @@ import { sanitizeType } from "./sanitize";
|
||||
import { IntoSql, toSQL } from "./util";
|
||||
export { IndexConfig } from "./native";
|
||||
export {
|
||||
BucketStats,
|
||||
GenerationStats,
|
||||
LsmStats,
|
||||
TableShardStats,
|
||||
SsTableStats,
|
||||
MemWalStats,
|
||||
MemtableStats,
|
||||
} from "./native";
|
||||
|
||||
@@ -217,7 +217,7 @@ export type TokenizeTableOptions =
|
||||
* `column` value, or upserts of that key can land in different shards and a
|
||||
* stale version can win).
|
||||
*/
|
||||
export interface LsmWriteSpec {
|
||||
export interface MemWalWriteSpec {
|
||||
/** One of `"bucket"`, `"identity"`, or `"unsharded"`. */
|
||||
specType: "bucket" | "identity" | "unsharded";
|
||||
/** Bucket and identity variants: the sharding column. */
|
||||
@@ -758,10 +758,10 @@ export abstract class Table {
|
||||
*/
|
||||
abstract setUnenforcedPrimaryKey(columns: string | string[]): Promise<void>;
|
||||
/**
|
||||
* Install an {@link LsmWriteSpec} on this table, selecting Lance's MemWAL
|
||||
* Install an {@link MemWalWriteSpec} on this table, selecting Lance's MemWAL
|
||||
* LSM-style write path for future `mergeInsert` calls.
|
||||
*
|
||||
* `LsmWriteSpec` chooses one of three sharding strategies via `specType`:
|
||||
* `MemWalWriteSpec` chooses one of three sharding strategies via `specType`:
|
||||
*
|
||||
* - `"bucket"` — hash-bucket writes by the single-column unenforced primary
|
||||
* key (`column` and `numBuckets` required).
|
||||
@@ -776,12 +776,12 @@ export abstract class Table {
|
||||
* including ones created later, and skips a kind the MemWAL cannot maintain.
|
||||
* Naming them pins an exact set, and a still-building index is rejected
|
||||
* rather than quietly omitted.
|
||||
* @param {LsmWriteSpec} spec The sharding spec to install.
|
||||
* @param {MemWalWriteSpec} spec The sharding spec to install.
|
||||
* @returns {Promise<void>}
|
||||
* @example
|
||||
* ```ts
|
||||
* await table.setUnenforcedPrimaryKey("id");
|
||||
* await table.setLsmWriteSpec({
|
||||
* await table.enableMemWal({
|
||||
* specType: "bucket",
|
||||
* column: "id",
|
||||
* numBuckets: 16,
|
||||
@@ -789,59 +789,59 @@ export abstract class Table {
|
||||
* });
|
||||
* ```
|
||||
*/
|
||||
abstract setLsmWriteSpec(spec: LsmWriteSpec): Promise<void>;
|
||||
abstract enableMemWal(spec: MemWalWriteSpec): Promise<void>;
|
||||
/**
|
||||
* Remove the {@link LsmWriteSpec} from this table, reverting to the standard
|
||||
* Remove the {@link MemWalWriteSpec} from this table, reverting to the standard
|
||||
* `mergeInsert` write path.
|
||||
*
|
||||
* Errors if no spec is currently set.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
abstract unsetLsmWriteSpec(): Promise<void>;
|
||||
abstract disableMemWal(): Promise<void>;
|
||||
/**
|
||||
* Read the {@link LsmWriteSpec} currently installed on this table.
|
||||
* Read the {@link MemWalWriteSpec} currently installed on this table.
|
||||
*
|
||||
* Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
|
||||
* spec has been set, or it was removed with {@link Table#unsetLsmWriteSpec}).
|
||||
* Resolves to `undefined` when the MemWAL write path is not enabled (no
|
||||
* spec has been set, or it was removed with {@link Table#disableMemWal}).
|
||||
* The spec is the one installed, including its maintained-index selection:
|
||||
* an absent `maintainedIndexes` for every index the table has, an empty
|
||||
* array for none.
|
||||
* @returns {Promise<LsmWriteSpec | undefined>}
|
||||
* @returns {Promise<MemWalWriteSpec | undefined>}
|
||||
*/
|
||||
abstract getLsmWriteSpec(): Promise<LsmWriteSpec | undefined>;
|
||||
abstract getMemWalSpec(): Promise<MemWalWriteSpec | undefined>;
|
||||
/**
|
||||
* Drain and close any cached MemWAL shard writers held for this table.
|
||||
*
|
||||
* When an {@link LsmWriteSpec} is installed, `mergeInsert` opens MemWAL
|
||||
* When an {@link MemWalWriteSpec} is installed, `mergeInsert` opens MemWAL
|
||||
* shard writers and caches them for reuse across calls. This closes them,
|
||||
* flushing pending data; writers reopen lazily on the next `mergeInsert`.
|
||||
* It is a no-op when no writers are cached.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
abstract closeLsmWriters(): Promise<void>;
|
||||
abstract closeMemWalWriters(): Promise<void>;
|
||||
/**
|
||||
* Seal every bucket's active memtable into a new L0 generation.
|
||||
* Freeze every table shard's active memtable into a new SSTable.
|
||||
*
|
||||
* Returns once the seal is committed. Sealing an empty memtable is a no-op,
|
||||
* Returns once the freeze is committed. Freezing an empty memtable is a no-op,
|
||||
* so this is safe to call repeatedly.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
abstract flushLsm(): Promise<void>;
|
||||
abstract flushMemWal(): Promise<void>;
|
||||
/**
|
||||
* Trigger a background L0 → base compaction pass per bucket.
|
||||
* Trigger a background SSTable compaction pass per table shard.
|
||||
*
|
||||
* Returns once the passes are *dispatched*, not once they finish — watch
|
||||
* {@link Table#getLsmStats} for progress, or use
|
||||
* {@link Table#checkpointLsm} to wait for convergence.
|
||||
* {@link Table#getMemWalStats} for progress, or use
|
||||
* {@link Table#checkpointMemWal} to wait for convergence.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
abstract compactLsm(): Promise<void>;
|
||||
abstract compactMemWal(): Promise<void>;
|
||||
/**
|
||||
* Converge this table's LSM write path into its base table.
|
||||
* Converge this table's MemWAL write path into its base table.
|
||||
*
|
||||
* Seals once, then triggers compaction and polls until the L0 that existed
|
||||
* Freezes once, then triggers compaction and polls until the SSTables that existed
|
||||
* at the start is gone. The target set is fixed at the start, so
|
||||
* generations created *during* the checkpoint are ignored — that is what
|
||||
* SSTables created *during* the checkpoint are ignored — that is what
|
||||
* lets it terminate under write load, and what makes it best-effort: it
|
||||
* converges the fresh tier as of some instant. Idempotent, abandonable at
|
||||
* any point, and safe to run on a cadence.
|
||||
@@ -852,26 +852,26 @@ export abstract class Table {
|
||||
* @returns {Promise<void>}
|
||||
* @example
|
||||
* ```ts
|
||||
* const before = await table.getLsmStats();
|
||||
* await table.checkpointLsm();
|
||||
* const after = await table.getLsmStats();
|
||||
* const before = await table.getMemWalStats();
|
||||
* await table.checkpointMemWal();
|
||||
* const after = await table.getMemWalStats();
|
||||
* ```
|
||||
*/
|
||||
abstract checkpointLsm(): Promise<void>;
|
||||
abstract checkpointMemWal(): Promise<void>;
|
||||
/**
|
||||
* Read live per-bucket LSM state.
|
||||
* Read live per-table-shard MemWAL state.
|
||||
*
|
||||
* Answers "how far behind is my fresh tier", "which bucket is hot", and
|
||||
* Answers "how far behind is my fresh tier", "which table shard is hot", and
|
||||
* "why is my fresh-tier vector search brute-force". Mutates no table state.
|
||||
*
|
||||
* Resolves to `undefined` only when the LSM write path is not enabled.
|
||||
* @param {boolean} includeGenerationRows Also count rows per L0 generation.
|
||||
* Resolves to `undefined` only when the MemWAL write path is not enabled.
|
||||
* @param {boolean} includeSstableRows Also count rows per SSTable.
|
||||
* Off by default because each count opens an uncached Lance dataset.
|
||||
* @returns {Promise<LsmStats | undefined>}
|
||||
* @returns {Promise<MemWalStats | undefined>}
|
||||
*/
|
||||
abstract getLsmStats(
|
||||
includeGenerationRows?: boolean,
|
||||
): Promise<LsmStats | undefined>;
|
||||
abstract getMemWalStats(
|
||||
includeSstableRows?: boolean,
|
||||
): Promise<MemWalStats | undefined>;
|
||||
/** Retrieve the version of the table */
|
||||
|
||||
abstract version(): Promise<number>;
|
||||
@@ -1474,43 +1474,43 @@ export class LocalTable extends Table {
|
||||
return await this.inner.setUnenforcedPrimaryKey(cols);
|
||||
}
|
||||
|
||||
async setLsmWriteSpec(spec: LsmWriteSpec): Promise<void> {
|
||||
return await this.inner.setLsmWriteSpec(spec);
|
||||
async enableMemWal(spec: MemWalWriteSpec): Promise<void> {
|
||||
return await this.inner.enableMemWal(spec);
|
||||
}
|
||||
|
||||
async unsetLsmWriteSpec(): Promise<void> {
|
||||
return await this.inner.unsetLsmWriteSpec();
|
||||
async disableMemWal(): Promise<void> {
|
||||
return await this.inner.disableMemWal();
|
||||
}
|
||||
|
||||
async getLsmWriteSpec(): Promise<LsmWriteSpec | undefined> {
|
||||
async getMemWalSpec(): Promise<MemWalWriteSpec | undefined> {
|
||||
// The native binding types `specType` as a plain `string`; narrow it back
|
||||
// to the public union. The Rust `From` impl only ever emits one of the
|
||||
// three valid values, so the cast is safe.
|
||||
return ((await this.inner.getLsmWriteSpec()) ?? undefined) as
|
||||
| LsmWriteSpec
|
||||
return ((await this.inner.getMemWalSpec()) ?? undefined) as
|
||||
| MemWalWriteSpec
|
||||
| undefined;
|
||||
}
|
||||
|
||||
async closeLsmWriters(): Promise<void> {
|
||||
return await this.inner.closeLsmWriters();
|
||||
async closeMemWalWriters(): Promise<void> {
|
||||
return await this.inner.closeMemWalWriters();
|
||||
}
|
||||
|
||||
async flushLsm(): Promise<void> {
|
||||
return await this.inner.flushLsm();
|
||||
async flushMemWal(): Promise<void> {
|
||||
return await this.inner.flushMemWal();
|
||||
}
|
||||
|
||||
async compactLsm(): Promise<void> {
|
||||
return await this.inner.compactLsm();
|
||||
async compactMemWal(): Promise<void> {
|
||||
return await this.inner.compactMemWal();
|
||||
}
|
||||
|
||||
async checkpointLsm(): Promise<void> {
|
||||
return await this.inner.checkpointLsm();
|
||||
async checkpointMemWal(): Promise<void> {
|
||||
return await this.inner.checkpointMemWal();
|
||||
}
|
||||
|
||||
async getLsmStats(
|
||||
includeGenerationRows: boolean = false,
|
||||
): Promise<LsmStats | undefined> {
|
||||
return (await this.inner.getLsmStats(includeGenerationRows)) ?? undefined;
|
||||
async getMemWalStats(
|
||||
includeSstableRows: boolean = false,
|
||||
): Promise<MemWalStats | undefined> {
|
||||
return (await this.inner.getMemWalStats(includeSstableRows)) ?? undefined;
|
||||
}
|
||||
|
||||
async version(): Promise<number> {
|
||||
|
||||
Reference in New Issue
Block a user