mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-04 12:38:38 +00:00
refactor(lsm): rename LSM stats to SSTable and table shard
Aligns the LSM stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an SSTable. Two terms in this API predate that pass. **Generation -> SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket -> table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding transform, so it was wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `LsmStats.buckets` | `LsmStats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Applied across Rust, Python, TypeScript, and Java, including the `get_lsm_stats` JSON field names. `LsmWriteSpec::Bucket` and `num_buckets` are unchanged — those name the sharding transform, not the shard. ## Compatibility Breaking for the MemWAL LSM stats API, which is experimental and paired with a server that renames the same fields. The JSON keys `table_shards` and `sstables` must roll out together with the WAL server change. ## Validation - `cargo check -p lancedb --all-features`, `cargo fmt --all` - `ruff format --check` and `ruff check` on the touched Python - `biome check` on the touched TypeScript The typedoc markdown under `docs/src/js` was updated by hand, not regenerated: `npm run docs` needs the napi-built `./native` types. Worth running `npm run docs` on this branch to confirm the generator agrees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgVf5C2yRbwK6pe1aMYkmg
This commit is contained in:
@@ -221,7 +221,7 @@ abstract checkpointLsm(): Promise<void>
|
||||
|
||||
Converge this table's LSM write path into its base table.
|
||||
|
||||
Seals once, then triggers compaction and polls until the L0 that existed
|
||||
Freezes once, then triggers compaction and polls until the SSTables that existed
|
||||
at the start is gone. The target set is fixed at the start, so
|
||||
generations created *during* the checkpoint are ignored — that is what
|
||||
lets it terminate under write load, and what makes it best-effort: it
|
||||
@@ -289,7 +289,7 @@ It is a no-op when no writers are cached.
|
||||
abstract compactLsm(): Promise<void>
|
||||
```
|
||||
|
||||
Trigger a background L0 → base compaction pass per bucket.
|
||||
Trigger a background SSTable compaction pass per table shard.
|
||||
|
||||
Returns once the passes are *dispatched*, not once they finish — watch
|
||||
[Table#getLsmStats](Table.md#getlsmstats) for progress, or use
|
||||
@@ -505,7 +505,7 @@ Drop an index from the table.
|
||||
abstract flushLsm(): Promise<void>
|
||||
```
|
||||
|
||||
Seal every bucket's active memtable into a new L0 generation.
|
||||
Freeze every table shard's active memtable into a new SSTable.
|
||||
|
||||
Returns once the seal is committed. Sealing an empty memtable is a no-op,
|
||||
so this is safe to call repeatedly.
|
||||
@@ -519,10 +519,10 @@ so this is safe to call repeatedly.
|
||||
### getLsmStats()
|
||||
|
||||
```ts
|
||||
abstract getLsmStats(includeGenerationRows?): Promise<undefined | LsmStats>
|
||||
abstract getLsmStats(includeSstableRows?): Promise<undefined | LsmStats>
|
||||
```
|
||||
|
||||
Read live per-bucket LSM state.
|
||||
Read live per-table-shard LSM state.
|
||||
|
||||
Answers "how far behind is my fresh tier", "which bucket is hot", and
|
||||
"why is my fresh-tier vector search brute-force". Mutates no table state.
|
||||
@@ -531,8 +531,8 @@ Resolves to `undefined` only when the LSM write path is not enabled.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **includeGenerationRows?**: `boolean`
|
||||
Also count rows per L0 generation.
|
||||
* **includeSstableRows?**: `boolean`
|
||||
Also count rows per SSTable.
|
||||
Off by default because each count opens an uncached Lance dataset.
|
||||
|
||||
#### Returns
|
||||
|
||||
@@ -60,7 +60,6 @@
|
||||
- [BranchDiff](interfaces/BranchDiff.md)
|
||||
- [BranchIndexSummary](interfaces/BranchIndexSummary.md)
|
||||
- [BranchRowCountSummary](interfaces/BranchRowCountSummary.md)
|
||||
- [BucketStats](interfaces/BucketStats.md)
|
||||
- [CherryPickError](interfaces/CherryPickError.md)
|
||||
- [CherryPickPreview](interfaces/CherryPickPreview.md)
|
||||
- [CherryPickResult](interfaces/CherryPickResult.md)
|
||||
@@ -87,7 +86,6 @@
|
||||
- [FtsToken](interfaces/FtsToken.md)
|
||||
- [FullTextQuery](interfaces/FullTextQuery.md)
|
||||
- [FullTextSearchOptions](interfaces/FullTextSearchOptions.md)
|
||||
- [GenerationStats](interfaces/GenerationStats.md)
|
||||
- [HnswPqOptions](interfaces/HnswPqOptions.md)
|
||||
- [HnswSqOptions](interfaces/HnswSqOptions.md)
|
||||
- [IndexConfig](interfaces/IndexConfig.md)
|
||||
@@ -126,7 +124,9 @@
|
||||
- [SplitHashOptions](interfaces/SplitHashOptions.md)
|
||||
- [SplitRandomOptions](interfaces/SplitRandomOptions.md)
|
||||
- [SplitSequentialOptions](interfaces/SplitSequentialOptions.md)
|
||||
- [SsTableStats](interfaces/SsTableStats.md)
|
||||
- [TableNamesOptions](interfaces/TableNamesOptions.md)
|
||||
- [TableShardStats](interfaces/TableShardStats.md)
|
||||
- [TableStatistics](interfaces/TableStatistics.md)
|
||||
- [TimeoutConfig](interfaces/TimeoutConfig.md)
|
||||
- [TlsConfig](interfaces/TlsConfig.md)
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / GenerationStats
|
||||
|
||||
# Interface: GenerationStats
|
||||
|
||||
One flushed L0 generation.
|
||||
|
||||
## Properties
|
||||
|
||||
### bytes
|
||||
|
||||
```ts
|
||||
bytes: number;
|
||||
```
|
||||
|
||||
On-disk size of the generation.
|
||||
|
||||
***
|
||||
|
||||
### generation
|
||||
|
||||
```ts
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation number. Increases as memtables are sealed into L0.
|
||||
|
||||
***
|
||||
|
||||
### rows?
|
||||
|
||||
```ts
|
||||
optional rows: number;
|
||||
```
|
||||
|
||||
Present only when `includeGenerationRows` was requested. Off by default
|
||||
because each count opens an uncached Lance dataset.
|
||||
@@ -6,17 +6,17 @@
|
||||
|
||||
# Interface: LsmStats
|
||||
|
||||
Live per-bucket LSM state, as returned by `Table#getLsmStats`.
|
||||
Live per-table-shard LSM state, as returned by `Table#getLsmStats`.
|
||||
|
||||
Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are
|
||||
Nothing here is derived: sums and differences (total SSTable bytes, WAL lag) are
|
||||
the caller's to compute.
|
||||
|
||||
## Properties
|
||||
|
||||
### buckets
|
||||
### tableShards
|
||||
|
||||
```ts
|
||||
buckets: BucketStats[];
|
||||
tableShards: TableShardStats[];
|
||||
```
|
||||
|
||||
One entry per bucket backing this table.
|
||||
One entry per table shard backing this table.
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / SsTableStats
|
||||
|
||||
# Interface: SsTableStats
|
||||
|
||||
One SSTable.
|
||||
|
||||
## Properties
|
||||
|
||||
### bytes
|
||||
|
||||
```ts
|
||||
bytes: number;
|
||||
```
|
||||
|
||||
On-disk size of the SSTable.
|
||||
|
||||
***
|
||||
|
||||
### generation
|
||||
|
||||
```ts
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation number. Increases as memtables are frozen into SSTables.
|
||||
|
||||
***
|
||||
|
||||
### rows?
|
||||
|
||||
```ts
|
||||
optional rows: number;
|
||||
```
|
||||
|
||||
Present only when `includeSstableRows` was requested. Off by default
|
||||
because each count opens an uncached Lance dataset.
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BucketStats
|
||||
[@lancedb/lancedb](../globals.md) / TableShardStats
|
||||
|
||||
# Interface: BucketStats
|
||||
# Interface: TableShardStats
|
||||
|
||||
Live state of one bucket. A table is N buckets on one node; flattening to a
|
||||
single number hides the one hot bucket that is usually why someone opened
|
||||
Live state of one table shard. A table is N table shards on one node; flattening to a
|
||||
single number hides the one hot table shard that is usually why someone opened
|
||||
this endpoint.
|
||||
|
||||
## Properties
|
||||
@@ -18,7 +18,7 @@ this endpoint.
|
||||
compacting: boolean;
|
||||
```
|
||||
|
||||
Whether a pass owns this bucket's compaction latch right now. Says *a*
|
||||
Whether a pass owns this table shard's compaction latch right now. Says *a*
|
||||
driver is running, not *whose*, and the latch is held from dispatch —
|
||||
including while the pass queues for a pod-wide compactor permit. Read it
|
||||
as "do not pile on", never as "mine is progressing".
|
||||
@@ -35,13 +35,13 @@ The generation the active memtable will become.
|
||||
|
||||
***
|
||||
|
||||
### generations
|
||||
### sstables
|
||||
|
||||
```ts
|
||||
generations: GenerationStats[];
|
||||
sstables: SsTableStats[];
|
||||
```
|
||||
|
||||
Flushed L0 generations not yet merged into the base table.
|
||||
SSTables not yet merged into the base table.
|
||||
|
||||
***
|
||||
|
||||
@@ -61,7 +61,7 @@ Version of the shard manifest these numbers were read from.
|
||||
optional memtables: MemtableStats[];
|
||||
```
|
||||
|
||||
Oldest first, active last. Absent for a `"Sealed"` bucket, whose
|
||||
Oldest first, active last. Absent for a `"Sealed"` table shard, whose
|
||||
in-memory state is torn down.
|
||||
|
||||
***
|
||||
@@ -82,7 +82,7 @@ WAL position replay resumes from.
|
||||
shardId: string;
|
||||
```
|
||||
|
||||
The shard this bucket writes.
|
||||
The shard this table shard writes.
|
||||
|
||||
***
|
||||
|
||||
Reference in New Issue
Block a user