refactor(lsm): rename LSM stats to SSTable and table shard

Aligns the LSM stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
SSTable. Two terms in this API predate that pass.

**Generation -> SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.

**Bucket -> table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding transform, so it was wrong for `identity` and
`year` sharding, which produce shards and no buckets at all.

| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `LsmStats.buckets` | `LsmStats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |

Applied across Rust, Python, TypeScript, and Java, including the
`get_lsm_stats` JSON field names. `LsmWriteSpec::Bucket` and `num_buckets`
are unchanged — those name the sharding transform, not the shard.

## Compatibility

Breaking for the MemWAL LSM stats API, which is experimental and paired
with a server that renames the same fields. The JSON keys `table_shards`
and `sstables` must roll out together with the WAL server change.

## Validation

- `cargo check -p lancedb --all-features`, `cargo fmt --all`
- `ruff format --check` and `ruff check` on the touched Python
- `biome check` on the touched TypeScript

The typedoc markdown under `docs/src/js` was updated by hand, not
regenerated: `npm run docs` needs the napi-built `./native` types. Worth
running `npm run docs` on this branch to confirm the generator agrees.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgVf5C2yRbwK6pe1aMYkmg
This commit is contained in:
Daniel Rammer
2026-08-27 15:18:13 -05:00
co-authored by Claude Opus 5
parent 9d3962686e
commit ab74aa620c
24 changed files with 323 additions and 317 deletions
-40
View File
@@ -1,40 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / GenerationStats
# Interface: GenerationStats
One flushed L0 generation.
## Properties
### bytes
```ts
bytes: number;
```
On-disk size of the generation.
***
### generation
```ts
generation: number;
```
The generation number. Increases as memtables are sealed into L0.
***
### rows?
```ts
optional rows: number;
```
Present only when `includeGenerationRows` was requested. Off by default
because each count opens an uncached Lance dataset.
+5 -5
View File
@@ -6,17 +6,17 @@
# Interface: LsmStats
Live per-bucket LSM state, as returned by `Table#getLsmStats`.
Live per-table-shard LSM state, as returned by `Table#getLsmStats`.
Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are
Nothing here is derived: sums and differences (total SSTable bytes, WAL lag) are
the caller's to compute.
## Properties
### buckets
### tableShards
```ts
buckets: BucketStats[];
tableShards: TableShardStats[];
```
One entry per bucket backing this table.
One entry per table shard backing this table.
+40
View File
@@ -0,0 +1,40 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / SsTableStats
# Interface: SsTableStats
One SSTable.
## Properties
### bytes
```ts
bytes: number;
```
On-disk size of the SSTable.
***
### generation
```ts
generation: number;
```
The generation number. Increases as memtables are frozen into SSTables.
***
### rows?
```ts
optional rows: number;
```
Present only when `includeSstableRows` was requested. Off by default
because each count opens an uncached Lance dataset.
@@ -2,12 +2,12 @@
***
[@lancedb/lancedb](../globals.md) / BucketStats
[@lancedb/lancedb](../globals.md) / TableShardStats
# Interface: BucketStats
# Interface: TableShardStats
Live state of one bucket. A table is N buckets on one node; flattening to a
single number hides the one hot bucket that is usually why someone opened
Live state of one table shard. A table is N table shards on one node; flattening to a
single number hides the one hot table shard that is usually why someone opened
this endpoint.
## Properties
@@ -18,7 +18,7 @@ this endpoint.
compacting: boolean;
```
Whether a pass owns this bucket's compaction latch right now. Says *a*
Whether a pass owns this table shard's compaction latch right now. Says *a*
driver is running, not *whose*, and the latch is held from dispatch —
including while the pass queues for a pod-wide compactor permit. Read it
as "do not pile on", never as "mine is progressing".
@@ -35,13 +35,13 @@ The generation the active memtable will become.
***
### generations
### sstables
```ts
generations: GenerationStats[];
sstables: SsTableStats[];
```
Flushed L0 generations not yet merged into the base table.
SSTables not yet merged into the base table.
***
@@ -61,7 +61,7 @@ Version of the shard manifest these numbers were read from.
optional memtables: MemtableStats[];
```
Oldest first, active last. Absent for a `"Sealed"` bucket, whose
Oldest first, active last. Absent for a `"Sealed"` table shard, whose
in-memory state is torn down.
***
@@ -82,7 +82,7 @@ WAL position replay resumes from.
shardId: string;
```
The shard this bucket writes.
The shard this table shard writes.
***