refactor(lsm): rename LSM stats to SSTable and table shard

Aligns the LSM stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
SSTable. Two terms in this API predate that pass.

**Generation -> SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.

**Bucket -> table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding transform, so it was wrong for `identity` and
`year` sharding, which produce shards and no buckets at all.

| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `LsmStats.buckets` | `LsmStats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |

Applied across Rust, Python, TypeScript, and Java, including the
`get_lsm_stats` JSON field names. `LsmWriteSpec::Bucket` and `num_buckets`
are unchanged — those name the sharding transform, not the shard.

## Compatibility

Breaking for the MemWAL LSM stats API, which is experimental and paired
with a server that renames the same fields. The JSON keys `table_shards`
and `sstables` must roll out together with the WAL server change.

## Validation

- `cargo check -p lancedb --all-features`, `cargo fmt --all`
- `ruff format --check` and `ruff check` on the touched Python
- `biome check` on the touched TypeScript

The typedoc markdown under `docs/src/js` was updated by hand, not
regenerated: `npm run docs` needs the napi-built `./native` types. Worth
running `npm run docs` on this branch to confirm the generator agrees.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgVf5C2yRbwK6pe1aMYkmg
This commit is contained in:
Daniel Rammer
2026-08-27 15:14:08 -05:00
parent 9d3962686e
commit ab74aa620c
24 changed files with 323 additions and 317 deletions
@@ -26,7 +26,7 @@ import java.util.OptionalLong;
*
* <p>Installing an {@link LsmWriteSpec} routes {@code mergeInsert} upserts through Lance's MemWAL —
* an LSM-style append — instead of the standard merge path. Rows land in an in-memory memtable,
* seal into L0 generations, and are merged into the base table by compaction.
* freeze into SSTables, and are merged into the base table by compaction.
*
* <p>These routes are not part of the Lance Namespace specification, so they are issued directly
* rather than through {@link org.lance.namespace.LanceNamespace}.
@@ -38,7 +38,7 @@ import java.util.OptionalLong;
* .buildRestClient();
*
* LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");
* lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));
* lsm.setLsmWriteSpec(LsmWriteSpec.tableShard("id", 16));
* // ... merge_insert traffic ...
* lsm.checkpointLsm();
* }</pre>
@@ -94,7 +94,7 @@ public class LanceDbTableLsm {
* Install an {@link LsmWriteSpec} on this table, selecting the MemWAL LSM write path for future
* {@code mergeInsert} calls.
*
* <p>All variants require the table to have an unenforced primary key; bucket sharding
* <p>All variants require the table to have an unenforced primary key; tableShard sharding
* additionally requires it to be the single column being bucketed.
*/
public void setLsmWriteSpec(LsmWriteSpec spec) {
@@ -130,7 +130,7 @@ public class LanceDbTableLsm {
}
/**
* Seal every bucket's active memtable into a new L0 generation.
* Freeze every table shard's active memtable into a new SSTable.
*
* <p>Returns once the seal is committed. Sealing an empty memtable is a no-op, so this is safe to
* call repeatedly.
@@ -140,7 +140,7 @@ public class LanceDbTableLsm {
}
/**
* Trigger a background L0 → base compaction pass per bucket.
* Trigger a background SSTable compaction pass per table shard.
*
* <p>Returns once the passes are <em>dispatched</em>, not once they finish — watch {@link
* #getLsmStats}, or use {@link #checkpointLsm} to wait for convergence.
@@ -150,9 +150,9 @@ public class LanceDbTableLsm {
}
/**
* Read live per-bucket LSM state.
* Read live per-tableShard LSM state.
*
* <p>Answers "how far behind is my fresh tier", "which bucket is hot", and "why is my fresh-tier
* <p>Answers "how far behind is my fresh tier", "which tableShard is hot", and "why is my fresh-tier
* vector search brute-force". Mutates no table state.
*
* <p>Empty only when the LSM write path is not enabled — that is, when the server sends an absent
@@ -160,13 +160,13 @@ public class LanceDbTableLsm {
* one throws rather than decoding to something empty, because {@link #checkpointLsm} reads
* convergence out of these numbers and cannot tell a defaulted array from a drained one.
*
* @param includeGenerationRows Also count rows per L0 generation. Off by default because each
* @param includeSstableRows Also count rows per SSTable. Off by default because each
* count opens an uncached Lance dataset.
* @throws IllegalStateException if the response is absent or does not decode.
*/
public Optional<LsmStats> getLsmStats(boolean includeGenerationRows) {
public Optional<LsmStats> getLsmStats(boolean includeSstableRows) {
Map<String, Object> body = new LinkedHashMap<String, Object>();
body.put("include_generation_rows", includeGenerationRows);
body.put("include_sstable_rows", includeSstableRows);
JsonNode response = client.post(route("get_lsm_stats"), body);
if (response == null) {
throw new IllegalStateException("get_lsm_stats returned an empty response body");
@@ -186,8 +186,8 @@ public class LanceDbTableLsm {
/**
* Converge this table's LSM write path into its base table.
*
* <p>Seals once, fixes a target watermark from the resulting L0, then triggers compaction and
* polls until that L0 is gone. The target set is fixed at the start, so generations created
* <p>Freezes once, fixes a target watermark from the resulting SSTables, then triggers compaction and
* polls until those SSTables are gone. The target set is fixed at the start, so sstables created
* <em>during</em> the checkpoint are ignored — that is what lets it terminate under write load,
* and what makes it best-effort: it converges the fresh tier as of some instant. Idempotent,
* abandonable at any point, safe on a cadence.
@@ -204,7 +204,7 @@ public class LanceDbTableLsm {
for (int reissue = 0; reissue <= MAX_REISSUES; reissue++) {
// The seal turns everything written before this call into a generation, so the
// watermark has to be read after it. Idempotent: sealing an empty memtable is a
// no-op, so a re-issue does not churn empty generations.
// no-op, so a re-issue does not churn empty sstables.
if (issueVoid(this::flushLsm)) {
backoff(reissue);
continue;
@@ -220,7 +220,7 @@ public class LanceDbTableLsm {
return;
}
Map<String, Long> targets = newestGenerations(stats.value.get());
Map<String, Long> targets = newestSstableGenerations(stats.value.get());
if (targets.isEmpty()) {
return;
}
@@ -236,7 +236,7 @@ public class LanceDbTableLsm {
}
/**
* Trigger and poll until no bucket holds a generation at or below its target.
* Trigger and poll until no tableShard holds a generation at or below its target.
*
* @return true when the drain finished, false when the table needs re-claiming from flush.
*/
@@ -250,21 +250,21 @@ public class LanceDbTableLsm {
return true;
}
// `compacting` is the bucket's compaction latch, held from dispatch until the pass
// `compacting` is the tableShard's compaction latch, held from dispatch until the pass
// ends — including while it waits on a pod-wide permit. So it answers one question
// only: do not pile on. Buckets with nothing outstanding are skipped, not counted
// as idle.
long outstanding = 0;
boolean allCompacting = true;
for (BucketStats bucket : stats.value.get().buckets()) {
Long target = targets.get(bucket.shardId());
for (TableShardStats tableShard : stats.value.get().tableShards()) {
Long target = targets.get(tableShard.shardId());
if (target == null) {
continue;
}
long remaining = bucket.outstandingGenerations(target);
long remaining = tableShard.outstandingSstables(target);
if (remaining > 0) {
outstanding += remaining;
allCompacting &= bucket.compacting();
allCompacting &= tableShard.compacting();
}
}
if (outstanding == 0) {
@@ -281,7 +281,7 @@ public class LanceDbTableLsm {
if (!isRetryable(e)) {
throw e;
}
// A 429 here means the server could latch no bucket at all, which the poll
// A 429 here means the server could latch no tableShard at all, which the poll
// above already handles. Not retried in place: the latch it would contend for
// is the one doing the work, so fall through and re-read — POLL_INTERVAL_MS is
// the backoff.
@@ -291,13 +291,13 @@ public class LanceDbTableLsm {
}
}
/** The newest generation held by each bucket, skipping buckets holding none. */
private static Map<String, Long> newestGenerations(LsmStats stats) {
/** The newest generation held by each tableShard, skipping tableShards holding none. */
private static Map<String, Long> newestSstableGenerations(LsmStats stats) {
Map<String, Long> targets = new HashMap<String, Long>();
for (BucketStats bucket : stats.buckets()) {
OptionalLong newest = bucket.newestGeneration();
for (TableShardStats tableShard : stats.tableShards()) {
OptionalLong newest = tableShard.newestSstableGeneration();
if (newest.isPresent()) {
targets.put(bucket.shardId(), newest.getAsLong());
targets.put(tableShard.shardId(), newest.getAsLong());
}
}
return targets;
@@ -20,37 +20,37 @@ import java.util.Collections;
import java.util.List;
/**
* Live per-bucket LSM state, as returned by {@link LanceDbTableLsm#getLsmStats()}.
* Live per-tableShard LSM state, as returned by {@link LanceDbTableLsm#getLsmStats()}.
*
* <p>Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are the caller's to
* <p>Nothing here is derived: sums and differences (total SSTable bytes, WAL lag) are the caller's to
* compute. There is no "LSM is off" shape — that case is an empty {@link java.util.Optional},
* because a stats object of zeros would read as measurements.
*/
public class LsmStats {
private static final String CONTEXT = "lsm stats";
private final List<BucketStats> buckets;
private final List<TableShardStats> tableShards;
LsmStats(List<BucketStats> buckets) {
this.buckets = Collections.unmodifiableList(buckets);
LsmStats(List<TableShardStats> tableShards) {
this.tableShards = Collections.unmodifiableList(tableShards);
}
/** One entry per bucket. */
public List<BucketStats> buckets() {
return buckets;
/** One entry per tableShard. */
public List<TableShardStats> tableShards() {
return tableShards;
}
static LsmStats fromJson(JsonNode node) {
JsonFields.requiredObject(node, CONTEXT);
List<BucketStats> buckets = new ArrayList<BucketStats>();
for (JsonNode bucket : JsonFields.requiredArray(node, "buckets", CONTEXT)) {
buckets.add(BucketStats.fromJson(bucket));
List<TableShardStats> tableShards = new ArrayList<TableShardStats>();
for (JsonNode tableShard : JsonFields.requiredArray(node, "table_shards", CONTEXT)) {
tableShards.add(TableShardStats.fromJson(tableShard));
}
return new LsmStats(buckets);
return new LsmStats(tableShards);
}
@Override
public String toString() {
return "LsmStats{buckets=" + buckets + "}";
return "LsmStats{tableShards=" + tableShards + "}";
}
}
@@ -17,21 +17,21 @@ import com.fasterxml.jackson.databind.JsonNode;
import java.util.OptionalLong;
/** One flushed L0 generation. */
public class GenerationStats {
/** One SSTable. */
public class SsTableStats {
private static final String CONTEXT = "generation stats";
private final long generation;
private final long bytes;
private final Long rows;
GenerationStats(long generation, long bytes, Long rows) {
SsTableStats(long generation, long bytes, Long rows) {
this.generation = generation;
this.bytes = bytes;
this.rows = rows;
}
/** The generation number. Increases as memtables are sealed into L0. */
/** The generation number. Increases as memtables are frozen into SSTables. */
public long generation() {
return generation;
}
@@ -42,16 +42,16 @@ public class GenerationStats {
}
/**
* Rows in this generation, present only when {@code includeGenerationRows} was requested. Off by
* Rows in this generation, present only when {@code includeSstableRows} was requested. Off by
* default because each count opens an uncached Lance dataset.
*/
public OptionalLong rows() {
return rows == null ? OptionalLong.empty() : OptionalLong.of(rows);
}
static GenerationStats fromJson(JsonNode node) {
static SsTableStats fromJson(JsonNode node) {
JsonFields.requiredObject(node, CONTEXT);
return new GenerationStats(
return new SsTableStats(
JsonFields.requiredLong(node, "generation", CONTEXT),
JsonFields.requiredLong(node, "bytes", CONTEXT),
JsonFields.optionalLong(node, "rows", CONTEXT));
@@ -59,6 +59,6 @@ public class GenerationStats {
@Override
public String toString() {
return "GenerationStats{generation=" + generation + ", bytes=" + bytes + ", rows=" + rows + "}";
return "SsTableStats{generation=" + generation + ", bytes=" + bytes + ", rows=" + rows + "}";
}
}
@@ -22,11 +22,11 @@ import java.util.Optional;
import java.util.OptionalLong;
/**
* Live state of one bucket. A table is N buckets on one node; flattening to a single number hides
* the one hot bucket that is usually why someone opened this endpoint.
* Live state of one tableShard. A table is N tableShards on one node; flattening to a single number hides
* the one hot tableShard that is usually why someone opened this endpoint.
*/
public class BucketStats {
private static final String CONTEXT = "bucket stats";
public class TableShardStats {
private static final String CONTEXT = "tableShard stats";
private final String shardId;
private final String status;
@@ -35,11 +35,11 @@ public class BucketStats {
private final long currentGeneration;
private final long replayAfterWalEntryPosition;
private final long walEntryPositionLastSeen;
private final List<GenerationStats> generations;
private final List<SsTableStats> sstables;
private final boolean compacting;
private final List<MemtableStats> memtables;
BucketStats(
TableShardStats(
String shardId,
String status,
long writerEpoch,
@@ -47,7 +47,7 @@ public class BucketStats {
long currentGeneration,
long replayAfterWalEntryPosition,
long walEntryPositionLastSeen,
List<GenerationStats> generations,
List<SsTableStats> sstables,
boolean compacting,
List<MemtableStats> memtables) {
this.shardId = shardId;
@@ -57,12 +57,12 @@ public class BucketStats {
this.currentGeneration = currentGeneration;
this.replayAfterWalEntryPosition = replayAfterWalEntryPosition;
this.walEntryPositionLastSeen = walEntryPositionLastSeen;
this.generations = Collections.unmodifiableList(generations);
this.sstables = Collections.unmodifiableList(sstables);
this.compacting = compacting;
this.memtables = memtables == null ? null : Collections.unmodifiableList(memtables);
}
/** The shard this bucket writes. */
/** The shard this tableShard writes. */
public String shardId() {
return shardId;
}
@@ -100,13 +100,13 @@ public class BucketStats {
return walEntryPositionLastSeen;
}
/** Flushed L0 generations not yet merged into the base table. */
public List<GenerationStats> generations() {
return generations;
/** SSTables not yet merged into the base table. */
public List<SsTableStats> sstables() {
return sstables;
}
/**
* Whether a pass owns this bucket's compaction latch right now. Says <em>a</em> driver is
* Whether a pass owns this tableShard's compaction latch right now. Says <em>a</em> driver is
* running, not <em>whose</em>, and the latch is held from dispatch including while the pass
* queues for a pod-wide compactor permit. Read it as "do not pile on", never as "mine is
* progressing".
@@ -115,15 +115,15 @@ public class BucketStats {
return compacting;
}
/** Oldest first, active last. Empty for a {@code "Sealed"} bucket, whose state is torn down. */
/** Oldest first, active last. Empty for a {@code "Sealed"} tableShard, whose state is torn down. */
public Optional<List<MemtableStats>> memtables() {
return Optional.ofNullable(memtables);
}
/** The newest flushed generation, or empty when L0 is empty. */
OptionalLong newestGeneration() {
/** The newest SSTable generation, or empty when the tier is empty. */
OptionalLong newestSstableGeneration() {
OptionalLong newest = OptionalLong.empty();
for (GenerationStats generation : generations) {
for (SsTableStats generation : sstables) {
if (!newest.isPresent() || generation.generation() > newest.getAsLong()) {
newest = OptionalLong.of(generation.generation());
}
@@ -132,15 +132,15 @@ public class BucketStats {
}
/**
* How many generations at or below {@code target} are still in L0.
* How many SSTables at or below {@code target} are still uncompacted.
*
* <p>A count, not a boolean: one pass drains a bounded prefix rather than the whole target set,
* so a boolean would read as "no progress" for every pass but the last. Compaction drains
* oldest-first, so this decreases monotonically.
*/
long outstandingGenerations(long target) {
long outstandingSstables(long target) {
long count = 0;
for (GenerationStats generation : generations) {
for (SsTableStats generation : sstables) {
if (generation.generation() <= target) {
count++;
}
@@ -148,11 +148,11 @@ public class BucketStats {
return count;
}
static BucketStats fromJson(JsonNode node) {
static TableShardStats fromJson(JsonNode node) {
JsonFields.requiredObject(node, CONTEXT);
List<GenerationStats> generations = new ArrayList<GenerationStats>();
for (JsonNode generation : JsonFields.requiredArray(node, "generations", CONTEXT)) {
generations.add(GenerationStats.fromJson(generation));
List<SsTableStats> sstables = new ArrayList<SsTableStats>();
for (JsonNode generation : JsonFields.requiredArray(node, "sstables", CONTEXT)) {
sstables.add(SsTableStats.fromJson(generation));
}
JsonNode memtablesNode = JsonFields.optionalArray(node, "memtables", CONTEXT);
@@ -164,7 +164,7 @@ public class BucketStats {
}
}
return new BucketStats(
return new TableShardStats(
JsonFields.requiredText(node, "shard_id", CONTEXT),
JsonFields.requiredText(node, "status", CONTEXT),
JsonFields.requiredLong(node, "writer_epoch", CONTEXT),
@@ -172,21 +172,21 @@ public class BucketStats {
JsonFields.requiredLong(node, "current_generation", CONTEXT),
JsonFields.requiredLong(node, "replay_after_wal_entry_position", CONTEXT),
JsonFields.requiredLong(node, "wal_entry_position_last_seen", CONTEXT),
generations,
sstables,
JsonFields.requiredBoolean(node, "compacting", CONTEXT),
memtables);
}
@Override
public String toString() {
return "BucketStats{shardId="
return "TableShardStats{shardId="
+ shardId
+ ", status="
+ status
+ ", currentGeneration="
+ currentGeneration
+ ", generations="
+ generations
+ ", sstables="
+ sstables
+ ", compacting="
+ compacting
+ "}";
@@ -132,10 +132,10 @@ public class LanceDbTableLsmTest {
enqueue("set_lsm_write_spec", 200, "");
lsm.setLsmWriteSpec(
LsmWriteSpec.bucket("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
LsmWriteSpec.tableShard("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
JsonNode body = MAPPER.readTree(requestBodies.get(0));
assertEquals("bucket", body.get("sharding").get("mode").asText());
assertEquals("tableShard", body.get("sharding").get("mode").asText());
assertEquals("id", body.get("sharding").get("column").asText());
assertEquals(16, body.get("sharding").get("num_buckets").asInt());
assertEquals(1, body.get("maintained_indexes").size());
@@ -201,7 +201,7 @@ public class LanceDbTableLsmTest {
enqueue(
"get_lsm_write_spec",
200,
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"bucket\",\"column\":\"id\","
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"tableShard\",\"column\":\"id\","
+ "\"num_buckets\":16},\"maintained_indexes\":[\"id_idx\"],"
+ "\"writer_config_defaults\":{\"durable_write\":\"true\"}}}");
@@ -228,14 +228,14 @@ public class LanceDbTableLsmTest {
@Test
public void testGetLsmStats() throws Exception {
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
Optional<LsmStats> got = lsm.getLsmStats(true);
assertEquals("/v1/table/my_table/get_lsm_stats/", requestPaths.get(0));
assertTrue(MAPPER.readTree(requestBodies.get(0)).get("include_generation_rows").asBoolean());
assertTrue(MAPPER.readTree(requestBodies.get(0)).get("include_sstable_rows").asBoolean());
assertTrue(got.isPresent());
BucketStats decoded = got.get().buckets().get(0);
TableShardStats decoded = got.get().tableShards().get(0);
assertEquals("shard-0", decoded.shardId());
assertEquals("Active", decoded.status());
assertEquals(1, decoded.writerEpoch());
@@ -243,8 +243,8 @@ public class LanceDbTableLsmTest {
assertEquals(9, decoded.currentGeneration());
assertFalse(decoded.compacting());
assertEquals(Arrays.asList(7L, 8L), generationNumbers(decoded));
assertEquals(1024, decoded.generations().get(0).bytes());
assertFalse(decoded.generations().get(0).rows().isPresent(), "rows absent unless requested");
assertEquals(1024, decoded.sstables().get(0).bytes());
assertFalse(decoded.sstables().get(0).rows().isPresent(), "rows absent unless requested");
assertFalse(decoded.memtables().isPresent(), "absent memtables stay absent");
}
@@ -254,19 +254,19 @@ public class LanceDbTableLsmTest {
enqueue(
"get_lsm_stats",
200,
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
"{\"lsm_stats\":{\"tableShards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
+ "\"replay_after_wal_entry_position\":3,\"wal_entry_position_last_seen\":11,"
+ "\"generations\":[{\"generation\":7,\"bytes\":1024,\"rows\":42}],"
+ "\"sstables\":[{\"generation\":7,\"bytes\":1024,\"rows\":42}],"
+ "\"compacting\":true,\"memtables\":[{\"generation\":8,\"rows\":5,"
+ "\"bytes\":64,\"batches\":2,\"indexes\":[\"id_idx\"]}]}]}}");
BucketStats decoded = lsm.getLsmStats(true).get().buckets().get(0);
TableShardStats decoded = lsm.getLsmStats(true).get().tableShards().get(0);
assertEquals(3, decoded.replayAfterWalEntryPosition());
assertEquals(11, decoded.walEntryPositionLastSeen());
assertTrue(decoded.compacting());
assertEquals(42, decoded.generations().get(0).rows().getAsLong());
assertEquals(42, decoded.sstables().get(0).rows().getAsLong());
assertTrue(decoded.memtables().isPresent());
MemtableStats memtable = decoded.memtables().get().get(0);
assertEquals(8, memtable.generation());
@@ -289,7 +289,7 @@ public class LanceDbTableLsmTest {
lsm.getLsmStats();
assertFalse(MAPPER.readTree(requestBodies.get(0)).get("include_generation_rows").asBoolean());
assertFalse(MAPPER.readTree(requestBodies.get(0)).get("include_sstable_rows").asBoolean());
}
// ===========================================================================
@@ -334,8 +334,8 @@ public class LanceDbTableLsmTest {
@Test
public void testCheckpointReturnsWhenNoGenerationsOutstanding() {
enqueue("flush_lsm", 200, "");
// A bucket with no L0 generations yields no target, so the drain never starts.
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
// A table shard with no SSTables yields no target, so the drain never starts.
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false)));
lsm.checkpointLsm();
@@ -345,12 +345,12 @@ public class LanceDbTableLsmTest {
@Test
public void testCheckpointConvergesOnceTargetGenerationsAreGone() {
enqueue("flush_lsm", 200, "");
// Watermark read: shard-0 holds generations 7 and 8, so target = 8.
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
// Watermark read: shard-0 holds sstables 7 and 8, so target = 8.
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
// First drain poll: both still outstanding, nothing compacting -> dispatch a pass.
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
// Second drain poll: drained past the target -> done.
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 9L)));
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 9L)));
enqueue("compact_lsm", 200, "");
lsm.checkpointLsm();
@@ -362,14 +362,14 @@ public class LanceDbTableLsmTest {
@Test
public void testCheckpointDoesNotPileOnWhileEveryTargetBucketIsCompacting() {
enqueue("flush_lsm", 200, "");
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", true, 4L)));
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", true, 4L)));
// Still compacting on the first poll, so no pass is dispatched; then it drains.
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", true, 4L)));
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 5L)));
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", true, 4L)));
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 5L)));
lsm.checkpointLsm();
assertEquals(0, countCalls("compact_lsm"), "a latched bucket is left alone");
assertEquals(0, countCalls("compact_lsm"), "a latched tableShard is left alone");
}
@Test
@@ -378,7 +378,7 @@ public class LanceDbTableLsmTest {
// from flush rather than retrying the read in place.
enqueue("flush_lsm", 200, "");
enqueue("get_lsm_stats", 421, "no claim");
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false)));
lsm.checkpointLsm();
@@ -389,7 +389,7 @@ public class LanceDbTableLsmTest {
public void testCheckpointRetriesRetryableStatusInPlace() {
enqueue("flush_lsm", 429, "latch held");
enqueue("flush_lsm", 200, "");
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false)));
lsm.checkpointLsm();
@@ -421,27 +421,27 @@ public class LanceDbTableLsmTest {
/**
* A stats payload that does not decode must fail closed. Every one of these bodies used to be
* read as "no buckets", which is indistinguishable from a drained table, so {@code checkpointLsm}
* read as "no tableShards", which is indistinguishable from a drained table, so {@code checkpointLsm}
* reported convergence for a checkpoint that never ran.
*/
@Test
public void testCheckpointRejectsMalformedStats() {
Map<String, String> malformed = new LinkedHashMap<String, String>();
malformed.put("no response body at all", "");
malformed.put("stats object with no buckets", "{\"lsm_stats\":{}}");
malformed.put("bucket missing its required fields", "{\"lsm_stats\":{\"buckets\":[{}]}}");
malformed.put("stats object with no tableShards", "{\"lsm_stats\":{}}");
malformed.put("tableShard missing its required fields", "{\"lsm_stats\":{\"tableShards\":[{}]}}");
malformed.put(
"bucket missing generations",
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
"tableShard missing sstables",
"{\"lsm_stats\":{\"tableShards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
+ "\"compacting\":false}]}}");
malformed.put(
"generation with a non-numeric generation number",
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
"{\"lsm_stats\":{\"tableShards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
+ "\"generations\":[{\"generation\":\"7\",\"bytes\":1024}],"
+ "\"sstables\":[{\"generation\":\"7\",\"bytes\":1024}],"
+ "\"compacting\":false}]}}");
for (Map.Entry<String, String> each : malformed.entrySet()) {
@@ -492,22 +492,22 @@ public class LanceDbTableLsmTest {
// harness
// ===========================================================================
private static List<Long> generationNumbers(BucketStats bucket) {
private static List<Long> generationNumbers(TableShardStats tableShard) {
List<Long> numbers = new ArrayList<Long>();
for (GenerationStats generation : bucket.generations()) {
for (SsTableStats generation : tableShard.sstables()) {
numbers.add(generation.generation());
}
return numbers;
}
/** Build an {@code lsm_stats} response body from bucket fragments. */
private static String stats(String... buckets) {
return "{\"lsm_stats\":{\"buckets\":[" + String.join(",", buckets) + "]}}";
/** Build an {@code lsm_stats} response body from tableShard fragments. */
private static String stats(String... tableShards) {
return "{\"lsm_stats\":{\"tableShards\":[" + String.join(",", tableShards) + "]}}";
}
private static String bucket(String shardId, boolean compacting, Long... generations) {
private static String tableShard(String shardId, boolean compacting, Long... sstables) {
StringBuilder gens = new StringBuilder();
for (Long generation : generations) {
for (Long generation : sstables) {
if (gens.length() > 0) {
gens.append(",");
}
@@ -517,7 +517,7 @@ public class LanceDbTableLsmTest {
+ shardId
+ "\",\"status\":\"Active\",\"writer_epoch\":1,\"manifest_version\":2,"
+ "\"current_generation\":9,\"replay_after_wal_entry_position\":0,"
+ "\"wal_entry_position_last_seen\":0,\"generations\":["
+ "\"wal_entry_position_last_seen\":0,\"sstables\":["
+ gens
+ "],\"compacting\":"
+ compacting