mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-15 16:22:27 +00:00
refactor(lsm): rename LSM stats to SSTable and table shard
Aligns the LSM stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an SSTable. Two terms in this API predate that pass. **Generation -> SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket -> table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding transform, so it was wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `LsmStats.buckets` | `LsmStats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Applied across Rust, Python, TypeScript, and Java, including the `get_lsm_stats` JSON field names. `LsmWriteSpec::Bucket` and `num_buckets` are unchanged — those name the sharding transform, not the shard. ## Compatibility Breaking for the MemWAL LSM stats API, which is experimental and paired with a server that renames the same fields. The JSON keys `table_shards` and `sstables` must roll out together with the WAL server change. ## Validation - `cargo check -p lancedb --all-features`, `cargo fmt --all` - `ruff format --check` and `ruff check` on the touched Python - `biome check` on the touched TypeScript The typedoc markdown under `docs/src/js` was updated by hand, not regenerated: `npm run docs` needs the napi-built `./native` types. Worth running `npm run docs` on this branch to confirm the generator agrees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgVf5C2yRbwK6pe1aMYkmg
This commit is contained in:
co-authored by
Claude Opus 5
parent
9d3962686e
commit
ab74aa620c
@@ -1,40 +0,0 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / GenerationStats
|
||||
|
||||
# Interface: GenerationStats
|
||||
|
||||
One flushed L0 generation.
|
||||
|
||||
## Properties
|
||||
|
||||
### bytes
|
||||
|
||||
```ts
|
||||
bytes: number;
|
||||
```
|
||||
|
||||
On-disk size of the generation.
|
||||
|
||||
***
|
||||
|
||||
### generation
|
||||
|
||||
```ts
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation number. Increases as memtables are sealed into L0.
|
||||
|
||||
***
|
||||
|
||||
### rows?
|
||||
|
||||
```ts
|
||||
optional rows: number;
|
||||
```
|
||||
|
||||
Present only when `includeGenerationRows` was requested. Off by default
|
||||
because each count opens an uncached Lance dataset.
|
||||
@@ -6,17 +6,17 @@
|
||||
|
||||
# Interface: LsmStats
|
||||
|
||||
Live per-bucket LSM state, as returned by `Table#getLsmStats`.
|
||||
Live per-table-shard LSM state, as returned by `Table#getLsmStats`.
|
||||
|
||||
Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are
|
||||
Nothing here is derived: sums and differences (total SSTable bytes, WAL lag) are
|
||||
the caller's to compute.
|
||||
|
||||
## Properties
|
||||
|
||||
### buckets
|
||||
### tableShards
|
||||
|
||||
```ts
|
||||
buckets: BucketStats[];
|
||||
tableShards: TableShardStats[];
|
||||
```
|
||||
|
||||
One entry per bucket backing this table.
|
||||
One entry per table shard backing this table.
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / SsTableStats
|
||||
|
||||
# Interface: SsTableStats
|
||||
|
||||
One SSTable.
|
||||
|
||||
## Properties
|
||||
|
||||
### bytes
|
||||
|
||||
```ts
|
||||
bytes: number;
|
||||
```
|
||||
|
||||
On-disk size of the SSTable.
|
||||
|
||||
***
|
||||
|
||||
### generation
|
||||
|
||||
```ts
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation number. Increases as memtables are frozen into SSTables.
|
||||
|
||||
***
|
||||
|
||||
### rows?
|
||||
|
||||
```ts
|
||||
optional rows: number;
|
||||
```
|
||||
|
||||
Present only when `includeSstableRows` was requested. Off by default
|
||||
because each count opens an uncached Lance dataset.
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BucketStats
|
||||
[@lancedb/lancedb](../globals.md) / TableShardStats
|
||||
|
||||
# Interface: BucketStats
|
||||
# Interface: TableShardStats
|
||||
|
||||
Live state of one bucket. A table is N buckets on one node; flattening to a
|
||||
single number hides the one hot bucket that is usually why someone opened
|
||||
Live state of one table shard. A table is N table shards on one node; flattening to a
|
||||
single number hides the one hot table shard that is usually why someone opened
|
||||
this endpoint.
|
||||
|
||||
## Properties
|
||||
@@ -18,7 +18,7 @@ this endpoint.
|
||||
compacting: boolean;
|
||||
```
|
||||
|
||||
Whether a pass owns this bucket's compaction latch right now. Says *a*
|
||||
Whether a pass owns this table shard's compaction latch right now. Says *a*
|
||||
driver is running, not *whose*, and the latch is held from dispatch —
|
||||
including while the pass queues for a pod-wide compactor permit. Read it
|
||||
as "do not pile on", never as "mine is progressing".
|
||||
@@ -35,13 +35,13 @@ The generation the active memtable will become.
|
||||
|
||||
***
|
||||
|
||||
### generations
|
||||
### sstables
|
||||
|
||||
```ts
|
||||
generations: GenerationStats[];
|
||||
sstables: SsTableStats[];
|
||||
```
|
||||
|
||||
Flushed L0 generations not yet merged into the base table.
|
||||
SSTables not yet merged into the base table.
|
||||
|
||||
***
|
||||
|
||||
@@ -61,7 +61,7 @@ Version of the shard manifest these numbers were read from.
|
||||
optional memtables: MemtableStats[];
|
||||
```
|
||||
|
||||
Oldest first, active last. Absent for a `"Sealed"` bucket, whose
|
||||
Oldest first, active last. Absent for a `"Sealed"` table shard, whose
|
||||
in-memory state is torn down.
|
||||
|
||||
***
|
||||
@@ -82,7 +82,7 @@ WAL position replay resumes from.
|
||||
shardId: string;
|
||||
```
|
||||
|
||||
The shard this bucket writes.
|
||||
The shard this table shard writes.
|
||||
|
||||
***
|
||||
|
||||
Reference in New Issue
Block a user