mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
refactor!: rename the LSM table APIs to MemWAL
The write path these APIs drive is Lance's MemWAL, and "LSM" described its shape rather than naming it. The surface now says MemWAL, and turning it on and off reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. The flush/compact/checkpoint methods keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` and `optimize`, which compact the base table. ## Wire The remote routes and JSON envelopes move with the SDKs and must ship together with the matching server change: - `/set_lsm_write_spec/` -> `/enable_memwal/`, `/unset_lsm_write_spec/` -> `/disable_memwal/`, `/get_lsm_write_spec/` -> `/get_memwal_spec/`, `/flush_lsm/`, `/compact_lsm/`, `/get_lsm_stats/` -> `*_memwal/` - response keys `lsm_write_spec` -> `memwal_spec`, `lsm_stats` -> `memwal_stats`; query and merge-insert body field `use_lsm` -> `use_memwal` ## Also fixed The previous rename turned the Java `bucket` sharding mode into `tableShard` in the javadoc example and tests, and the test fixtures sent `tableShards` where the parser reads `table_shards`, so the Java tests did not compile. Both are restored. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` and `query::lsm` modules) are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel Rammer
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
06d7f82fa2
commit
d7e921dbec
@@ -425,16 +425,16 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): this
|
||||
useMemWal(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this query.
|
||||
|
||||
By default (unset), when the table carries a MemWAL write spec (see
|
||||
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
|
||||
[Table#enableMemWal](Table.md#enablememwal)), reads are routed through the LSM scanner so
|
||||
they also return data written via the `mergeInsert` LSM path that has not yet
|
||||
been compacted into the base table (the active/frozen in-memory memtables and
|
||||
the flushed generations), deduplicated by primary key; a table without a spec
|
||||
@@ -448,7 +448,7 @@ reads the base table.
|
||||
even when a spec is present.
|
||||
Note: the LSM scanner does not support every query shape (e.g. reranking,
|
||||
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
|
||||
`useLsm(false)` is set, because a base-only read would silently exclude
|
||||
`useMemWal(false)` is set, because a base-only read would silently exclude
|
||||
un-compacted MemWAL data.
|
||||
|
||||
#### Returns
|
||||
@@ -457,7 +457,7 @@ reads the base table.
|
||||
|
||||
#### Inherited from
|
||||
|
||||
`StandardQueryBase.useLsm`
|
||||
`StandardQueryBase.useMemWal`
|
||||
|
||||
***
|
||||
|
||||
|
||||
@@ -76,15 +76,15 @@ the query optimizer chooses a suboptimal path.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): MergeInsertBuilder
|
||||
useMemWal(enable): MergeInsertBuilder
|
||||
```
|
||||
|
||||
Control MemWAL routing for this merge.
|
||||
|
||||
By default (unset), a `mergeInsert` on a table with an LSM write spec is
|
||||
By default (unset), a `mergeInsert` on a table with a MemWAL write spec is
|
||||
routed through Lance's MemWAL shard writer, and a table without one uses the
|
||||
standard path.
|
||||
|
||||
@@ -92,7 +92,7 @@ standard path.
|
||||
|
||||
* **enable**: `boolean`
|
||||
`true` forces MemWAL routing and errors if the table has no
|
||||
LSM write spec. `false` forces the standard write path even when a spec is set.
|
||||
MemWAL write spec. `false` forces the standard write path even when a spec is set.
|
||||
|
||||
#### Returns
|
||||
|
||||
@@ -108,12 +108,12 @@ validateSingleShard(validateSingleShard): MergeInsertBuilder
|
||||
|
||||
Controls how an LSM merge checks that its input targets a single shard.
|
||||
|
||||
When a table has an LSM write spec, every row in a `mergeInsert` call must
|
||||
When a table has a MemWAL write spec, every row in a `mergeInsert` call must
|
||||
route to the same shard. When `true` (the default), every row is inspected
|
||||
to verify this. When `false`, only the first row is inspected and the
|
||||
shard it routes to is used for the whole input — a faster path for callers
|
||||
that have already pre-sharded their input. Has no effect on tables without
|
||||
an LSM write spec.
|
||||
a MemWAL write spec.
|
||||
|
||||
#### Parameters
|
||||
|
||||
|
||||
@@ -499,16 +499,16 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): this
|
||||
useMemWal(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this query.
|
||||
|
||||
By default (unset), when the table carries a MemWAL write spec (see
|
||||
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
|
||||
[Table#enableMemWal](Table.md#enablememwal)), reads are routed through the LSM scanner so
|
||||
they also return data written via the `mergeInsert` LSM path that has not yet
|
||||
been compacted into the base table (the active/frozen in-memory memtables and
|
||||
the flushed generations), deduplicated by primary key; a table without a spec
|
||||
@@ -522,7 +522,7 @@ reads the base table.
|
||||
even when a spec is present.
|
||||
Note: the LSM scanner does not support every query shape (e.g. reranking,
|
||||
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
|
||||
`useLsm(false)` is set, because a base-only read would silently exclude
|
||||
`useMemWal(false)` is set, because a base-only read would silently exclude
|
||||
un-compacted MemWAL data.
|
||||
|
||||
#### Returns
|
||||
@@ -531,7 +531,7 @@ reads the base table.
|
||||
|
||||
#### Inherited from
|
||||
|
||||
`StandardQueryBase.useLsm`
|
||||
`StandardQueryBase.useMemWal`
|
||||
|
||||
***
|
||||
|
||||
|
||||
@@ -227,13 +227,13 @@ version of the table.
|
||||
|
||||
***
|
||||
|
||||
### checkpointLsm()
|
||||
### checkpointMemWal()
|
||||
|
||||
```ts
|
||||
abstract checkpointLsm(): Promise<void>
|
||||
abstract checkpointMemWal(): Promise<void>
|
||||
```
|
||||
|
||||
Converge this table's LSM write path into its base table.
|
||||
Converge this table's MemWAL write path into its base table.
|
||||
|
||||
Freezes once, then triggers compaction and polls until the SSTables that existed
|
||||
at the start is gone. The target set is fixed at the start, so
|
||||
@@ -253,9 +253,9 @@ is merging. The caller owns the deadline.
|
||||
#### Example
|
||||
|
||||
```ts
|
||||
const before = await table.getLsmStats();
|
||||
await table.checkpointLsm();
|
||||
const after = await table.getLsmStats();
|
||||
const before = await table.getMemWalStats();
|
||||
await table.checkpointMemWal();
|
||||
const after = await table.getMemWalStats();
|
||||
```
|
||||
|
||||
***
|
||||
@@ -278,15 +278,15 @@ Any attempt to use the table after it is closed will result in an error.
|
||||
|
||||
***
|
||||
|
||||
### closeLsmWriters()
|
||||
### closeMemWalWriters()
|
||||
|
||||
```ts
|
||||
abstract closeLsmWriters(): Promise<void>
|
||||
abstract closeMemWalWriters(): Promise<void>
|
||||
```
|
||||
|
||||
Drain and close any cached MemWAL shard writers held for this table.
|
||||
|
||||
When an [LsmWriteSpec](../interfaces/LsmWriteSpec.md) is installed, `mergeInsert` opens MemWAL
|
||||
When an [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) is installed, `mergeInsert` opens MemWAL
|
||||
shard writers and caches them for reuse across calls. This closes them,
|
||||
flushing pending data; writers reopen lazily on the next `mergeInsert`.
|
||||
It is a no-op when no writers are cached.
|
||||
@@ -297,17 +297,17 @@ It is a no-op when no writers are cached.
|
||||
|
||||
***
|
||||
|
||||
### compactLsm()
|
||||
### compactMemWal()
|
||||
|
||||
```ts
|
||||
abstract compactLsm(): Promise<void>
|
||||
abstract compactMemWal(): Promise<void>
|
||||
```
|
||||
|
||||
Trigger a background SSTable compaction pass per table shard.
|
||||
|
||||
Returns once the passes are *dispatched*, not once they finish — watch
|
||||
[Table#getLsmStats](Table.md#getlsmstats) for progress, or use
|
||||
[Table#checkpointLsm](Table.md#checkpointlsm) to wait for convergence.
|
||||
[Table#getMemWalStats](Table.md#getmemwalstats) for progress, or use
|
||||
[Table#checkpointMemWal](Table.md#checkpointmemwal) to wait for convergence.
|
||||
|
||||
#### Returns
|
||||
|
||||
@@ -575,10 +575,10 @@ memory; use [Table.fetchBlobFiles](Table.md#fetchblobfiles) for large values.
|
||||
|
||||
***
|
||||
|
||||
### flushLsm()
|
||||
### flushMemWal()
|
||||
|
||||
```ts
|
||||
abstract flushLsm(): Promise<void>
|
||||
abstract flushMemWal(): Promise<void>
|
||||
```
|
||||
|
||||
Freeze every table shard's active memtable into a new SSTable.
|
||||
@@ -627,18 +627,18 @@ const { records, truncated } = await table.functionErrors({ column: "embedding"
|
||||
|
||||
***
|
||||
|
||||
### getLsmStats()
|
||||
### getMemWalStats()
|
||||
|
||||
```ts
|
||||
abstract getLsmStats(includeSstableRows?): Promise<undefined | LsmStats>
|
||||
abstract getMemWalStats(includeSstableRows?): Promise<undefined | MemWalStats>
|
||||
```
|
||||
|
||||
Read live per-table-shard LSM state.
|
||||
Read live per-table-shard MemWAL state.
|
||||
|
||||
Answers "how far behind is my fresh tier", "which bucket is hot", and
|
||||
"why is my fresh-tier vector search brute-force". Mutates no table state.
|
||||
|
||||
Resolves to `undefined` only when the LSM write path is not enabled.
|
||||
Resolves to `undefined` only when the MemWAL write path is not enabled.
|
||||
|
||||
#### Parameters
|
||||
|
||||
@@ -648,27 +648,27 @@ Resolves to `undefined` only when the LSM write path is not enabled.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`undefined` \| [`LsmStats`](../interfaces/LsmStats.md)>
|
||||
`Promise`<`undefined` \| [`MemWalStats`](../interfaces/MemWalStats.md)>
|
||||
|
||||
***
|
||||
|
||||
### getLsmWriteSpec()
|
||||
### getMemWalSpec()
|
||||
|
||||
```ts
|
||||
abstract getLsmWriteSpec(): Promise<undefined | LsmWriteSpec>
|
||||
abstract getMemWalSpec(): Promise<undefined | MemWalWriteSpec>
|
||||
```
|
||||
|
||||
Read the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) currently installed on this table.
|
||||
Read the [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) currently installed on this table.
|
||||
|
||||
Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
|
||||
spec has been set, or it was removed with [Table#unsetLsmWriteSpec](Table.md#unsetlsmwritespec)).
|
||||
Resolves to `undefined` when the MemWAL write path is not enabled (no
|
||||
spec has been set, or it was removed with [Table#disableMemWal](Table.md#disablememwal)).
|
||||
The spec is the one installed, including its maintained-index selection:
|
||||
an absent `maintainedIndexes` for every index the table has, an empty
|
||||
array for none.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`undefined` \| [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)>
|
||||
`Promise`<`undefined` \| [`MemWalWriteSpec`](../interfaces/MemWalWriteSpec.md)>
|
||||
|
||||
***
|
||||
|
||||
@@ -1086,16 +1086,16 @@ of the given query
|
||||
|
||||
***
|
||||
|
||||
### setLsmWriteSpec()
|
||||
### enableMemWal()
|
||||
|
||||
```ts
|
||||
abstract setLsmWriteSpec(spec): Promise<void>
|
||||
abstract enableMemWal(spec): Promise<void>
|
||||
```
|
||||
|
||||
Install an [LsmWriteSpec](../interfaces/LsmWriteSpec.md) on this table, selecting Lance's MemWAL
|
||||
Install an [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) on this table, selecting Lance's MemWAL
|
||||
LSM-style write path for future `mergeInsert` calls.
|
||||
|
||||
`LsmWriteSpec` chooses one of three sharding strategies via `specType`:
|
||||
`MemWalWriteSpec` chooses one of three sharding strategies via `specType`:
|
||||
|
||||
- `"bucket"` — hash-bucket writes by the single-column unenforced primary
|
||||
key (`column` and `numBuckets` required).
|
||||
@@ -1113,7 +1113,7 @@ rather than quietly omitted.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **spec**: [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)
|
||||
* **spec**: [`MemWalWriteSpec`](../interfaces/MemWalWriteSpec.md)
|
||||
The sharding spec to install.
|
||||
|
||||
#### Returns
|
||||
@@ -1124,7 +1124,7 @@ rather than quietly omitted.
|
||||
|
||||
```ts
|
||||
await table.setUnenforcedPrimaryKey("id");
|
||||
await table.setLsmWriteSpec({
|
||||
await table.enableMemWal({
|
||||
specType: "bucket",
|
||||
column: "id",
|
||||
numBuckets: 16,
|
||||
@@ -1288,13 +1288,13 @@ same tokenizer model files must also exist locally.
|
||||
|
||||
***
|
||||
|
||||
### unsetLsmWriteSpec()
|
||||
### disableMemWal()
|
||||
|
||||
```ts
|
||||
abstract unsetLsmWriteSpec(): Promise<void>
|
||||
abstract disableMemWal(): Promise<void>
|
||||
```
|
||||
|
||||
Remove the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) from this table, reverting to the standard
|
||||
Remove the [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) from this table, reverting to the standard
|
||||
`mergeInsert` write path.
|
||||
|
||||
Errors if no spec is currently set.
|
||||
|
||||
@@ -275,10 +275,10 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): this
|
||||
useMemWal(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this take query.
|
||||
|
||||
@@ -781,16 +781,16 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): this
|
||||
useMemWal(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this query.
|
||||
|
||||
By default (unset), when the table carries a MemWAL write spec (see
|
||||
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
|
||||
[Table#enableMemWal](Table.md#enablememwal)), reads are routed through the LSM scanner so
|
||||
they also return data written via the `mergeInsert` LSM path that has not yet
|
||||
been compacted into the base table (the active/frozen in-memory memtables and
|
||||
the flushed generations), deduplicated by primary key; a table without a spec
|
||||
@@ -804,7 +804,7 @@ reads the base table.
|
||||
even when a spec is present.
|
||||
Note: the LSM scanner does not support every query shape (e.g. reranking,
|
||||
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
|
||||
`useLsm(false)` is set, because a base-only read would silently exclude
|
||||
`useMemWal(false)` is set, because a base-only read would silently exclude
|
||||
un-compacted MemWAL data.
|
||||
|
||||
#### Returns
|
||||
@@ -813,7 +813,7 @@ reads the base table.
|
||||
|
||||
#### Inherited from
|
||||
|
||||
`StandardQueryBase.useLsm`
|
||||
`StandardQueryBase.useMemWal`
|
||||
|
||||
***
|
||||
|
||||
|
||||
Reference in New Issue
Block a user