mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
refactor(lsm): rename LSM stats to SSTable and table shard
Aligns the LSM stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an SSTable. Two terms in this API predate that pass. **Generation -> SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket -> table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding transform, so it was wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `LsmStats.buckets` | `LsmStats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Applied across Rust, Python, TypeScript, and Java, including the `get_lsm_stats` JSON field names. `LsmWriteSpec::Bucket` and `num_buckets` are unchanged — those name the sharding transform, not the shard. ## Compatibility Breaking for the MemWAL LSM stats API, which is experimental and paired with a server that renames the same fields. The JSON keys `table_shards` and `sstables` must roll out together with the WAL server change. ## Validation - `cargo check -p lancedb --all-features`, `cargo fmt --all` - `ruff format --check` and `ruff check` on the touched Python - `biome check` on the touched TypeScript The typedoc markdown under `docs/src/js` was updated by hand, not regenerated: `npm run docs` needs the napi-built `./native` types. Worth running `npm run docs` on this branch to confirm the generator agrees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgVf5C2yRbwK6pe1aMYkmg
This commit is contained in:
co-authored by
Claude Opus 5
parent
9d3962686e
commit
ab74aa620c
@@ -221,7 +221,7 @@ abstract checkpointLsm(): Promise<void>
|
||||
|
||||
Converge this table's LSM write path into its base table.
|
||||
|
||||
Seals once, then triggers compaction and polls until the L0 that existed
|
||||
Freezes once, then triggers compaction and polls until the SSTables that existed
|
||||
at the start is gone. The target set is fixed at the start, so
|
||||
generations created *during* the checkpoint are ignored — that is what
|
||||
lets it terminate under write load, and what makes it best-effort: it
|
||||
@@ -289,7 +289,7 @@ It is a no-op when no writers are cached.
|
||||
abstract compactLsm(): Promise<void>
|
||||
```
|
||||
|
||||
Trigger a background L0 → base compaction pass per bucket.
|
||||
Trigger a background SSTable compaction pass per table shard.
|
||||
|
||||
Returns once the passes are *dispatched*, not once they finish — watch
|
||||
[Table#getLsmStats](Table.md#getlsmstats) for progress, or use
|
||||
@@ -505,7 +505,7 @@ Drop an index from the table.
|
||||
abstract flushLsm(): Promise<void>
|
||||
```
|
||||
|
||||
Seal every bucket's active memtable into a new L0 generation.
|
||||
Freeze every table shard's active memtable into a new SSTable.
|
||||
|
||||
Returns once the seal is committed. Sealing an empty memtable is a no-op,
|
||||
so this is safe to call repeatedly.
|
||||
@@ -519,10 +519,10 @@ so this is safe to call repeatedly.
|
||||
### getLsmStats()
|
||||
|
||||
```ts
|
||||
abstract getLsmStats(includeGenerationRows?): Promise<undefined | LsmStats>
|
||||
abstract getLsmStats(includeSstableRows?): Promise<undefined | LsmStats>
|
||||
```
|
||||
|
||||
Read live per-bucket LSM state.
|
||||
Read live per-table-shard LSM state.
|
||||
|
||||
Answers "how far behind is my fresh tier", "which bucket is hot", and
|
||||
"why is my fresh-tier vector search brute-force". Mutates no table state.
|
||||
@@ -531,8 +531,8 @@ Resolves to `undefined` only when the LSM write path is not enabled.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **includeGenerationRows?**: `boolean`
|
||||
Also count rows per L0 generation.
|
||||
* **includeSstableRows?**: `boolean`
|
||||
Also count rows per SSTable.
|
||||
Off by default because each count opens an uncached Lance dataset.
|
||||
|
||||
#### Returns
|
||||
|
||||
Reference in New Issue
Block a user