mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-22 11:42:27 +00:00
refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
0968c21f4c
commit
fe34eb3549
+151
-148
@@ -46,12 +46,12 @@ import static org.junit.jupiter.api.Assertions.*;
|
||||
* <p>The wire assertions mirror the Rust mocked-endpoint tests in {@code
|
||||
* rust/lancedb/src/remote/table.rs}, which are the contract these routes have to match.
|
||||
*/
|
||||
public class LanceDbTableLsmTest {
|
||||
public class LanceDbTableMemWalTest {
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
|
||||
private HttpServer server;
|
||||
private LanceDbRestClient client;
|
||||
private LanceDbTableLsm lsm;
|
||||
private LanceDbTableMemWal memWal;
|
||||
|
||||
private final List<String> requestPaths = Collections.synchronizedList(new ArrayList<String>());
|
||||
private final List<String> requestBodies = Collections.synchronizedList(new ArrayList<String>());
|
||||
@@ -101,7 +101,7 @@ public class LanceDbTableLsmTest {
|
||||
.database("test-db")
|
||||
.endpoint("http://127.0.0.1:" + server.getAddress().getPort())
|
||||
.buildRestClient();
|
||||
lsm = new LanceDbTableLsm(client, "my_table");
|
||||
memWal = new LanceDbTableMemWal(client, "my_table");
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
@@ -115,12 +115,12 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecUnsharded() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalUnsharded() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded());
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded());
|
||||
|
||||
assertEquals("/v1/table/my_table/set_lsm_write_spec/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/enable_memwal/", requestPaths.get(0));
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("unsharded", body.get("sharding").get("mode").asText());
|
||||
assertFalse(body.get("sharding").has("column"));
|
||||
@@ -128,11 +128,11 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecBucket() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalBucket() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(
|
||||
LsmWriteSpec.bucket("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
|
||||
memWal.enableMemWal(
|
||||
MemWalWriteSpec.bucket("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("bucket", body.get("sharding").get("mode").asText());
|
||||
@@ -143,10 +143,10 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecIdentity() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalIdentity() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.identity("tenant"));
|
||||
memWal.enableMemWal(MemWalWriteSpec.identity("tenant"));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("identity", body.get("sharding").get("mode").asText());
|
||||
@@ -160,55 +160,55 @@ public class LanceDbTableLsmTest {
|
||||
*/
|
||||
@Test
|
||||
public void testMaintainedIndexesNullAndEmptyAreDistinctOnTheWire() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded());
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded());
|
||||
JsonNode fresh = MAPPER.readTree(requestBodies.get(0));
|
||||
assertTrue(fresh.has("maintained_indexes"), "the key must be present");
|
||||
assertTrue(fresh.get("maintained_indexes").isNull(), "a fresh spec sends null, not []");
|
||||
|
||||
lsm.setLsmWriteSpec(
|
||||
LsmWriteSpec.unsharded().withMaintainedIndexes(Collections.<String>emptyList()));
|
||||
memWal.enableMemWal(
|
||||
MemWalWriteSpec.unsharded().withMaintainedIndexes(Collections.<String>emptyList()));
|
||||
JsonNode none = MAPPER.readTree(requestBodies.get(1));
|
||||
assertTrue(none.get("maintained_indexes").isArray());
|
||||
assertEquals(0, none.get("maintained_indexes").size());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecWriterConfigDefaults() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalWriterConfigDefaults() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
Map<String, String> defaults = new HashMap<String, String>();
|
||||
defaults.put("max_memtable_rows", "50000");
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded().withWriterConfigDefaults(defaults));
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded().withWriterConfigDefaults(defaults));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("50000", body.get("writer_config_defaults").get("max_memtable_rows").asText());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testUnsetLsmWriteSpec() {
|
||||
enqueue("unset_lsm_write_spec", 200, "");
|
||||
public void testDisableMemWal() {
|
||||
enqueue("disable_memwal", 200, "");
|
||||
|
||||
lsm.unsetLsmWriteSpec();
|
||||
memWal.disableMemWal();
|
||||
|
||||
assertEquals("/v1/table/my_table/unset_lsm_write_spec/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/disable_memwal/", requestPaths.get(0));
|
||||
assertEquals("", requestBodies.get(0));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpec() {
|
||||
public void testGetMemWalSpec() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"bucket\",\"column\":\"id\","
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"bucket\",\"column\":\"id\","
|
||||
+ "\"num_buckets\":16},\"maintained_indexes\":[\"id_idx\"],"
|
||||
+ "\"writer_config_defaults\":{\"durable_write\":\"true\"}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertEquals(LsmWriteSpec.Sharding.BUCKET, spec.get().sharding());
|
||||
assertEquals(MemWalWriteSpec.Sharding.BUCKET, spec.get().sharding());
|
||||
assertEquals("id", spec.get().column());
|
||||
assertEquals(Integer.valueOf(16), spec.get().numBuckets());
|
||||
assertEquals(Arrays.asList("id_idx"), spec.get().maintainedIndexes());
|
||||
@@ -217,28 +217,28 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
/** A null selection, every index the table has, reads back as null rather than an empty list. */
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRoundTripsEveryIndexSelection() {
|
||||
public void testGetMemWalSpecRoundTripsEveryIndexSelection() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":null,\"writer_config_defaults\":{}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertNull(spec.get().maintainedIndexes());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRoundTripsNoIndexSelection() {
|
||||
public void testGetMemWalSpecRoundTripsNoIndexSelection() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":[],\"writer_config_defaults\":{}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertEquals(Collections.emptyList(), spec.get().maintainedIndexes());
|
||||
@@ -249,32 +249,32 @@ public class LanceDbTableLsmTest {
|
||||
* rejected: reading a string or an object as null would turn a malformed response into a policy.
|
||||
*/
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRejectsANonArrayMaintainedIndexes() {
|
||||
public void testGetMemWalSpecRejectsANonArrayMaintainedIndexes() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":\"id_idx\",\"writer_config_defaults\":{}}}");
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> lsm.getLsmWriteSpec());
|
||||
assertThrows(IllegalStateException.class, () -> memWal.getMemWalSpec());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRejectsNonStringIndexNames() {
|
||||
public void testGetMemWalSpecRejectsNonStringIndexNames() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":[7],\"writer_config_defaults\":{}}}");
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> lsm.getLsmWriteSpec());
|
||||
assertThrows(IllegalStateException.class, () -> memWal.getMemWalSpec());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecAbsent() {
|
||||
enqueue("get_lsm_write_spec", 200, "{\"lsm_write_spec\":null}");
|
||||
public void testGetMemWalSpecAbsent() {
|
||||
enqueue("get_memwal_spec", 200, "{\"memwal_spec\":null}");
|
||||
|
||||
assertFalse(lsm.getLsmWriteSpec().isPresent());
|
||||
assertFalse(memWal.getMemWalSpec().isPresent());
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -282,15 +282,15 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testGetLsmStats() throws Exception {
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
public void testGetMemWalStats() throws Exception {
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
|
||||
Optional<LsmStats> got = lsm.getLsmStats(true);
|
||||
Optional<MemWalStats> got = memWal.getMemWalStats(true);
|
||||
|
||||
assertEquals("/v1/table/my_table/get_lsm_stats/", requestPaths.get(0));
|
||||
assertTrue(MAPPER.readTree(requestBodies.get(0)).get("include_generation_rows").asBoolean());
|
||||
assertEquals("/v1/table/my_table/get_memwal_stats/", requestPaths.get(0));
|
||||
assertTrue(MAPPER.readTree(requestBodies.get(0)).get("include_sstable_rows").asBoolean());
|
||||
assertTrue(got.isPresent());
|
||||
BucketStats decoded = got.get().buckets().get(0);
|
||||
TableShardStats decoded = got.get().tableShards().get(0);
|
||||
assertEquals("shard-0", decoded.shardId());
|
||||
assertEquals("Active", decoded.status());
|
||||
assertEquals(1, decoded.writerEpoch());
|
||||
@@ -298,30 +298,30 @@ public class LanceDbTableLsmTest {
|
||||
assertEquals(9, decoded.currentGeneration());
|
||||
assertFalse(decoded.compacting());
|
||||
assertEquals(Arrays.asList(7L, 8L), generationNumbers(decoded));
|
||||
assertEquals(1024, decoded.generations().get(0).bytes());
|
||||
assertFalse(decoded.generations().get(0).rows().isPresent(), "rows absent unless requested");
|
||||
assertEquals(1024, decoded.sstables().get(0).bytes());
|
||||
assertFalse(decoded.sstables().get(0).rows().isPresent(), "rows absent unless requested");
|
||||
assertFalse(decoded.memtables().isPresent(), "absent memtables stay absent");
|
||||
}
|
||||
|
||||
/** The optional fields decode when the server does send them. */
|
||||
@Test
|
||||
public void testGetLsmStatsDecodesOptionalFields() {
|
||||
public void testGetMemWalStatsDecodesOptionalFields() {
|
||||
enqueue(
|
||||
"get_lsm_stats",
|
||||
"get_memwal_stats",
|
||||
200,
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":3,\"wal_entry_position_last_seen\":11,"
|
||||
+ "\"generations\":[{\"generation\":7,\"bytes\":1024,\"rows\":42}],"
|
||||
+ "\"sstables\":[{\"generation\":7,\"bytes\":1024,\"rows\":42}],"
|
||||
+ "\"compacting\":true,\"memtables\":[{\"generation\":8,\"rows\":5,"
|
||||
+ "\"bytes\":64,\"batches\":2,\"indexes\":[\"id_idx\"]}]}]}}");
|
||||
|
||||
BucketStats decoded = lsm.getLsmStats(true).get().buckets().get(0);
|
||||
TableShardStats decoded = memWal.getMemWalStats(true).get().tableShards().get(0);
|
||||
|
||||
assertEquals(3, decoded.replayAfterWalEntryPosition());
|
||||
assertEquals(11, decoded.walEntryPositionLastSeen());
|
||||
assertTrue(decoded.compacting());
|
||||
assertEquals(42, decoded.generations().get(0).rows().getAsLong());
|
||||
assertEquals(42, decoded.sstables().get(0).rows().getAsLong());
|
||||
assertTrue(decoded.memtables().isPresent());
|
||||
MemtableStats memtable = decoded.memtables().get().get(0);
|
||||
assertEquals(8, memtable.generation());
|
||||
@@ -332,19 +332,19 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmStatsAbsentWhenLsmDisabled() {
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
public void testGetMemWalStatsAbsentWhenMemWalDisabled() {
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
assertFalse(lsm.getLsmStats().isPresent());
|
||||
assertFalse(memWal.getMemWalStats().isPresent());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmStatsDefaultsToExcludingGenerationRows() throws Exception {
|
||||
enqueue("get_lsm_stats", 200, stats());
|
||||
public void testGetMemWalStatsDefaultsToExcludingGenerationRows() throws Exception {
|
||||
enqueue("get_memwal_stats", 200, stats());
|
||||
|
||||
lsm.getLsmStats();
|
||||
memWal.getMemWalStats();
|
||||
|
||||
assertFalse(MAPPER.readTree(requestBodies.get(0)).get("include_generation_rows").asBoolean());
|
||||
assertFalse(MAPPER.readTree(requestBodies.get(0)).get("include_sstable_rows").asBoolean());
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -353,22 +353,22 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
@Test
|
||||
public void testFlushAndCompactRoutes() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("compact_lsm", 200, "");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("compact_memwal", 200, "");
|
||||
|
||||
lsm.flushLsm();
|
||||
lsm.compactLsm();
|
||||
memWal.flushMemWal();
|
||||
memWal.compactMemWal();
|
||||
|
||||
assertEquals("/v1/table/my_table/flush_lsm/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/compact_lsm/", requestPaths.get(1));
|
||||
assertEquals("/v1/table/my_table/flush_memwal/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/compact_memwal/", requestPaths.get(1));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testHttpErrorCarriesStatus() {
|
||||
enqueue("flush_lsm", 404, "no such table");
|
||||
enqueue("flush_memwal", 404, "no such table");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.flushLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.flushMemWal());
|
||||
assertEquals(404, e.statusCode());
|
||||
}
|
||||
|
||||
@@ -377,97 +377,98 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testCheckpointReturnsWhenLsmDisabled() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
public void testCheckpointReturnsWhenMemWalDisabled() {
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"), "nothing to compact when the LSM path is off");
|
||||
assertEquals(0, countCalls("compact_memwal"), "nothing to compact when the LSM path is off");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointReturnsWhenNoGenerationsOutstanding() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
// A bucket with no L0 generations yields no target, so the drain never starts.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
// A table shard with no SSTables yields no target, so the drain never starts.
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"));
|
||||
assertEquals(0, countCalls("compact_memwal"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointConvergesOnceTargetGenerationsAreGone() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
// Watermark read: shard-0 holds generations 7 and 8, so target = 8.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
// Watermark read: shard-0 holds sstables 7 and 8, so target = 8.
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
// First drain poll: both still outstanding, nothing compacting -> dispatch a pass.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
// Second drain poll: drained past the target -> done.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 9L)));
|
||||
enqueue("compact_lsm", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 9L)));
|
||||
enqueue("compact_memwal", 200, "");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(1, countCalls("compact_lsm"), "one pass dispatched");
|
||||
assertEquals(3, countCalls("get_lsm_stats"), "watermark read plus two drain polls");
|
||||
assertEquals(1, countCalls("compact_memwal"), "one pass dispatched");
|
||||
assertEquals(3, countCalls("get_memwal_stats"), "watermark read plus two drain polls");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointDoesNotPileOnWhileEveryTargetBucketIsCompacting() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", true, 4L)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", true, 4L)));
|
||||
// Still compacting on the first poll, so no pass is dispatched; then it drains.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", true, 4L)));
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 5L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", true, 4L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 5L)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"), "a latched bucket is left alone");
|
||||
assertEquals(0, countCalls("compact_memwal"), "a latched table shard is left alone");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointRetriesFromFlushAfterLostClaim() {
|
||||
// 421 on the watermark read: the node lost its claim, so the whole thing restarts
|
||||
// from flush rather than retrying the read in place.
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 421, "no claim");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 421, "no claim");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(2, countCalls("flush_lsm"), "re-issued from flush");
|
||||
assertEquals(2, countCalls("flush_memwal"), "re-issued from flush");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointRetriesRetryableStatusInPlace() {
|
||||
enqueue("flush_lsm", 429, "latch held");
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
enqueue("flush_memwal", 429, "latch held");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(2, countCalls("flush_lsm"), "429 retried in place, not re-issued");
|
||||
assertEquals(2, countCalls("flush_memwal"), "429 retried in place, not re-issued");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointPropagatesTerminalStatus() {
|
||||
enqueue("flush_lsm", 400, "bad request");
|
||||
enqueue("flush_memwal", 400, "bad request");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.checkpointLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.checkpointMemWal());
|
||||
assertEquals(400, e.statusCode());
|
||||
assertEquals(1, countCalls("flush_lsm"), "a terminal status is not retried");
|
||||
assertEquals(1, countCalls("flush_memwal"), "a terminal status is not retried");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointGivesUpAfterRepeatedLostClaims() {
|
||||
enqueue("flush_lsm", 421, "no claim");
|
||||
enqueue("flush_memwal", 421, "no claim");
|
||||
|
||||
IllegalStateException e = assertThrows(IllegalStateException.class, () -> lsm.checkpointLsm());
|
||||
IllegalStateException e =
|
||||
assertThrows(IllegalStateException.class, () -> memWal.checkpointMemWal());
|
||||
assertTrue(e.getMessage().contains("kept losing its claim"), e.getMessage());
|
||||
assertEquals(4, countCalls("flush_lsm"), "the initial attempt plus MAX_REISSUES");
|
||||
assertEquals(4, countCalls("flush_memwal"), "the initial attempt plus MAX_REISSUES");
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -476,50 +477,51 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
/**
|
||||
* A stats payload that does not decode must fail closed. Every one of these bodies used to be
|
||||
* read as "no buckets", which is indistinguishable from a drained table, so {@code checkpointLsm}
|
||||
* reported convergence for a checkpoint that never ran.
|
||||
* read as "no table shards", which is indistinguishable from a drained table, so {@code
|
||||
* checkpointMemWal} reported convergence for a checkpoint that never ran.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckpointRejectsMalformedStats() {
|
||||
Map<String, String> malformed = new LinkedHashMap<String, String>();
|
||||
malformed.put("no response body at all", "");
|
||||
malformed.put("stats object with no buckets", "{\"lsm_stats\":{}}");
|
||||
malformed.put("bucket missing its required fields", "{\"lsm_stats\":{\"buckets\":[{}]}}");
|
||||
malformed.put("stats object with no table shards", "{\"memwal_stats\":{}}");
|
||||
malformed.put(
|
||||
"bucket missing generations",
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"table shard missing its required fields", "{\"memwal_stats\":{\"table_shards\":[{}]}}");
|
||||
malformed.put(
|
||||
"table shard missing sstables",
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
|
||||
+ "\"compacting\":false}]}}");
|
||||
malformed.put(
|
||||
"generation with a non-numeric generation number",
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
|
||||
+ "\"generations\":[{\"generation\":\"7\",\"bytes\":1024}],"
|
||||
+ "\"sstables\":[{\"generation\":\"7\",\"bytes\":1024}],"
|
||||
+ "\"compacting\":false}]}}");
|
||||
|
||||
for (Map.Entry<String, String> each : malformed.entrySet()) {
|
||||
setUpFresh();
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, each.getValue());
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, each.getValue());
|
||||
|
||||
assertThrows(
|
||||
IllegalStateException.class,
|
||||
() -> lsm.checkpointLsm(),
|
||||
() -> memWal.checkpointMemWal(),
|
||||
each.getKey() + " must not report convergence");
|
||||
}
|
||||
}
|
||||
|
||||
/** The one shape that legitimately means "this table has no LSM write path". */
|
||||
/** The one shape that legitimately means "this table has no MemWAL write path". */
|
||||
@Test
|
||||
public void testCheckpointTreatsNullStatsAsNotWalBacked() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(1, countCalls("get_lsm_stats"));
|
||||
assertEquals(1, countCalls("get_memwal_stats"));
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -529,40 +531,41 @@ public class LanceDbTableLsmTest {
|
||||
/**
|
||||
* The transport must not retry on the checkpoint loop's behalf. Apache HttpClient's default
|
||||
* strategy retries exactly 429 and 503 — the two statuses {@code isRetryable} owns — which
|
||||
* doubled every budget here and also retried {@code compact_lsm} in place, where the loop is
|
||||
* doubled every budget here and also retried {@code compact_memwal} in place, where the loop is
|
||||
* built to fall through to a fresh stats poll instead.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckpointRetryBudgetIsNotDoubledByTheTransport() {
|
||||
enqueue("flush_lsm", 429, "latch held");
|
||||
enqueue("flush_memwal", 429, "latch held");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.checkpointLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.checkpointMemWal());
|
||||
|
||||
assertEquals(429, e.statusCode(), "the exhausted budget propagates the last error as itself");
|
||||
assertEquals(9, countCalls("flush_lsm"), "the initial request plus MAX_RETRIES, and no more");
|
||||
assertEquals(
|
||||
9, countCalls("flush_memwal"), "the initial request plus MAX_RETRIES, and no more");
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// harness
|
||||
// ===========================================================================
|
||||
|
||||
private static List<Long> generationNumbers(BucketStats bucket) {
|
||||
private static List<Long> generationNumbers(TableShardStats tableShard) {
|
||||
List<Long> numbers = new ArrayList<Long>();
|
||||
for (GenerationStats generation : bucket.generations()) {
|
||||
for (SsTableStats generation : tableShard.sstables()) {
|
||||
numbers.add(generation.generation());
|
||||
}
|
||||
return numbers;
|
||||
}
|
||||
|
||||
/** Build an {@code lsm_stats} response body from bucket fragments. */
|
||||
private static String stats(String... buckets) {
|
||||
return "{\"lsm_stats\":{\"buckets\":[" + String.join(",", buckets) + "]}}";
|
||||
/** Build a {@code memwal_stats} response body from table shard fragments. */
|
||||
private static String stats(String... tableShards) {
|
||||
return "{\"memwal_stats\":{\"table_shards\":[" + String.join(",", tableShards) + "]}}";
|
||||
}
|
||||
|
||||
private static String bucket(String shardId, boolean compacting, Long... generations) {
|
||||
private static String tableShard(String shardId, boolean compacting, Long... sstables) {
|
||||
StringBuilder gens = new StringBuilder();
|
||||
for (Long generation : generations) {
|
||||
for (Long generation : sstables) {
|
||||
if (gens.length() > 0) {
|
||||
gens.append(",");
|
||||
}
|
||||
@@ -572,7 +575,7 @@ public class LanceDbTableLsmTest {
|
||||
+ shardId
|
||||
+ "\",\"status\":\"Active\",\"writer_epoch\":1,\"manifest_version\":2,"
|
||||
+ "\"current_generation\":9,\"replay_after_wal_entry_position\":0,"
|
||||
+ "\"wal_entry_position_last_seen\":0,\"generations\":["
|
||||
+ "\"wal_entry_position_last_seen\":0,\"sstables\":["
|
||||
+ gens
|
||||
+ "],\"compacting\":"
|
||||
+ compacting
|
||||
@@ -597,7 +600,7 @@ public class LanceDbTableLsmTest {
|
||||
return requestPaths.stream().filter(path -> operationOf(path).equals(operation)).count();
|
||||
}
|
||||
|
||||
/** {@code /v1/table/my_table/flush_lsm/} -> {@code flush_lsm}. */
|
||||
/** {@code /v1/table/my_table/flush_memwal/} -> {@code flush_memwal}. */
|
||||
private static String operationOf(String path) {
|
||||
String[] segments = path.split("/");
|
||||
return segments.length == 0 ? "" : segments[segments.length - 1];
|
||||
Reference in New Issue
Block a user