mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 07:42:28 +00:00
refactor!: rename the LSM table APIs to MemWAL
The write path these APIs drive is Lance's MemWAL, and "LSM" described its shape rather than naming it. The surface now says MemWAL, and turning it on and off reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. The flush/compact/checkpoint methods keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` and `optimize`, which compact the base table. ## Wire The remote routes and JSON envelopes move with the SDKs and must ship together with the matching server change: - `/set_lsm_write_spec/` -> `/enable_memwal/`, `/unset_lsm_write_spec/` -> `/disable_memwal/`, `/get_lsm_write_spec/` -> `/get_memwal_spec/`, `/flush_lsm/`, `/compact_lsm/`, `/get_lsm_stats/` -> `*_memwal/` - response keys `lsm_write_spec` -> `memwal_spec`, `lsm_stats` -> `memwal_stats`; query and merge-insert body field `use_lsm` -> `use_memwal` ## Also fixed The previous rename turned the Java `bucket` sharding mode into `tableShard` in the javadoc example and tests, and the test fixtures sent `tableShards` where the parser reads `table_shards`, so the Java tests did not compile. Both are restored. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` and `query::lsm` modules) are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel Rammer
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
06d7f82fa2
commit
d7e921dbec
+7
-7
@@ -29,7 +29,7 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.build();
|
||||
```
|
||||
|
||||
## MemWAL LSM write path
|
||||
## MemWAL write path
|
||||
|
||||
Most table operations reach LanceDB through the `LanceNamespace` above, which is
|
||||
generated from the Lance Namespace specification. The MemWAL LSM routes are not part
|
||||
@@ -37,26 +37,26 @@ of that specification, so they are issued through a separate client:
|
||||
|
||||
```java
|
||||
import com.lancedb.LanceDbRestClient;
|
||||
import com.lancedb.LanceDbTableLsm;
|
||||
import com.lancedb.LsmWriteSpec;
|
||||
import com.lancedb.LanceDbTableMemWal;
|
||||
import com.lancedb.MemWalWriteSpec;
|
||||
|
||||
LanceDbRestClient client = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.apiKey("your_lancedb_cloud_api_key")
|
||||
.database("your_database_name")
|
||||
.buildRestClient();
|
||||
|
||||
LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");
|
||||
LanceDbTableMemWal memWal = new LanceDbTableMemWal(client, "my_table");
|
||||
|
||||
// Route future merge_insert upserts through the MemWAL, hash-bucketed by `id`.
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));
|
||||
memWal.enableMemWal(MemWalWriteSpec.bucket("id", 16));
|
||||
|
||||
// ... merge_insert traffic ...
|
||||
|
||||
// Converge the fresh tier into the base table.
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
// Inspect live per-table-shard state.
|
||||
lsm.getLsmStats().ifPresent(stats -> stats.tableShards().forEach(tableShard ->
|
||||
memWal.getMemWalStats().ifPresent(stats -> stats.tableShards().forEach(tableShard ->
|
||||
System.out.println(tableShard.shardId() + ": " + tableShard.sstables().size() + " SSTables")));
|
||||
|
||||
client.close();
|
||||
|
||||
@@ -20,11 +20,11 @@ import com.fasterxml.jackson.databind.JsonNode;
|
||||
*
|
||||
* <p>Every reader fails closed: a missing, null, or wrong-typed field throws rather than
|
||||
* defaulting. That mirrors the serde decoding the Rust client applies to the same payloads in
|
||||
* {@code rust/lancedb/src/table/lsm_stats.rs}, where a required field has no default and a
|
||||
* {@code rust/lancedb/src/table/memwal_stats.rs}, where a required field has no default and a
|
||||
* malformed response is an error rather than a zero.
|
||||
*
|
||||
* <p>The alternative — Jackson's {@code path()}, which yields a missing node that reads as an empty
|
||||
* array or a zero — is unsafe here because {@link LanceDbTableLsm#checkpointLsm()} decides
|
||||
* array or a zero — is unsafe here because {@link LanceDbTableMemWal#checkpointMemWal()} decides
|
||||
* convergence from these numbers. A defaulted {@code sstables} array is indistinguishable from a
|
||||
* drained one, so a malformed response would report a checkpoint that never happened.
|
||||
*/
|
||||
|
||||
@@ -151,8 +151,8 @@ public class LanceDbNamespaceClientBuilder {
|
||||
* Build a {@link LanceDbRestClient} for the same endpoint.
|
||||
*
|
||||
* <p>Needed only for LanceDB routes that the Lance Namespace specification does not cover — the
|
||||
* MemWAL LSM write path, reached through {@link LanceDbTableLsm}. Every other table operation
|
||||
* belongs on the {@link LanceNamespace} from {@link #build()}.
|
||||
* MemWAL write path, reached through {@link LanceDbTableMemWal}. Every other table
|
||||
* operation belongs on the {@link LanceNamespace} from {@link #build()}.
|
||||
*
|
||||
* <p>The returned client owns an HTTP connection pool; close it when you are done with it.
|
||||
*
|
||||
|
||||
@@ -31,9 +31,9 @@ import java.io.UncheckedIOException;
|
||||
* specification does not cover.
|
||||
*
|
||||
* <p>Most table operations reach LanceDB through {@link org.lance.namespace.LanceNamespace}, which
|
||||
* is generated from the namespace spec. A handful of routes — the MemWAL LSM write path in
|
||||
* is generated from the namespace spec. A handful of routes — the MemWAL write path in
|
||||
* particular — are served by the same endpoint but are not part of that spec, so they are issued
|
||||
* directly here. See {@link LanceDbTableLsm}.
|
||||
* directly here. See {@link LanceDbTableMemWal}.
|
||||
*
|
||||
* <p>Obtain one from {@link LanceDbNamespaceClientBuilder#buildRestClient()}.
|
||||
*/
|
||||
@@ -50,8 +50,8 @@ public class LanceDbRestClient implements Closeable {
|
||||
this.apiKey = apiKey;
|
||||
this.database = database;
|
||||
// Automatic retries off, deliberately. The default strategy retries 429 and 503 —
|
||||
// exactly the two statuses LanceDbTableLsm.checkpointLsm() acts on — which would
|
||||
// silently double its explicit retry budget and would also retry compact_lsm in
|
||||
// exactly the two statuses LanceDbTableMemWal.checkpointMemWal() acts on — which would
|
||||
// silently double its explicit retry budget and would also retry compact_memwal in
|
||||
// place, where the loop is designed to fall through to a fresh stats poll instead.
|
||||
// The checkpoint loop owns the 421/429/503 transitions; the transport must not.
|
||||
this.http = HttpClients.custom().disableAutomaticRetries().build();
|
||||
@@ -98,8 +98,9 @@ public class LanceDbRestClient implements Closeable {
|
||||
/**
|
||||
* A non-2xx response.
|
||||
*
|
||||
* <p>The status is exposed because callers act on it: {@link LanceDbTableLsm#checkpointLsm()}
|
||||
* treats 429 and 503 as retryable and 421 as a lost node claim.
|
||||
* <p>The status is exposed because callers act on it: {@link
|
||||
* LanceDbTableMemWal#checkpointMemWal()} treats 429 and 503 as retryable and 421 as a lost node
|
||||
* claim.
|
||||
*/
|
||||
public static class HttpException extends RuntimeException {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
+71
-71
@@ -22,11 +22,11 @@ import java.util.Optional;
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/**
|
||||
* The MemWAL LSM write path for one LanceDB Cloud or Enterprise table.
|
||||
* The MemWAL write path for one LanceDB Cloud or Enterprise table.
|
||||
*
|
||||
* <p>Installing an {@link LsmWriteSpec} routes {@code mergeInsert} upserts through Lance's MemWAL —
|
||||
* an LSM-style append — instead of the standard merge path. Rows land in an in-memory memtable,
|
||||
* freeze into SSTables, and are merged into the base table by compaction.
|
||||
* <p>Installing an {@link MemWalWriteSpec} routes {@code mergeInsert} upserts through Lance's
|
||||
* MemWAL — an LSM-style append — instead of the standard merge path. Rows land in an in-memory
|
||||
* memtable, freeze into SSTables, and are merged into the base table by compaction.
|
||||
*
|
||||
* <p>These routes are not part of the Lance Namespace specification, so they are issued directly
|
||||
* rather than through {@link org.lance.namespace.LanceNamespace}.
|
||||
@@ -37,23 +37,23 @@ import java.util.OptionalLong;
|
||||
* .database("your_database_name")
|
||||
* .buildRestClient();
|
||||
*
|
||||
* LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");
|
||||
* lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));
|
||||
* LanceDbTableMemWal memWal = new LanceDbTableMemWal(client, "my_table");
|
||||
* memWal.enableMemWal(MemWalWriteSpec.bucket("id", 16));
|
||||
* // ... merge_insert traffic ...
|
||||
* lsm.checkpointLsm();
|
||||
* memWal.checkpointMemWal();
|
||||
* }</pre>
|
||||
*/
|
||||
public class LanceDbTableLsm {
|
||||
public class LanceDbTableMemWal {
|
||||
|
||||
/**
|
||||
* Interval between {@code get_lsm_stats} polls during a checkpoint. One interval is roughly one
|
||||
* compaction pass, the granularity at which the answer can change.
|
||||
* Interval between {@code get_memwal_stats} polls during a checkpoint. One interval is roughly
|
||||
* one compaction pass, the granularity at which the answer can change.
|
||||
*/
|
||||
private static final long POLL_INTERVAL_MS = 5_000L;
|
||||
|
||||
/**
|
||||
* Cap on re-issues from {@code flushLsm} after a 421, so a crash-looping node cannot turn flush →
|
||||
* compact → 421 → flush into a spin.
|
||||
* Cap on re-issues from {@code flushMemWal} after a 421, so a crash-looping node cannot turn
|
||||
* flush → compact → 421 → flush into a spin.
|
||||
*
|
||||
* <p>Deliberately not shared with {@link #MAX_RETRIES}: a claim that keeps evaporating is a
|
||||
* broken node, while contention is routine and wants a real budget.
|
||||
@@ -79,7 +79,7 @@ public class LanceDbTableLsm {
|
||||
* @param tableIdentifier The table's full identifier, {@code $}-delimited when it sits inside a
|
||||
* namespace, such as {@code analytics$events}.
|
||||
*/
|
||||
public LanceDbTableLsm(LanceDbRestClient client, String tableIdentifier) {
|
||||
public LanceDbTableMemWal(LanceDbRestClient client, String tableIdentifier) {
|
||||
if (client == null) {
|
||||
throw new IllegalArgumentException("Client cannot be null");
|
||||
}
|
||||
@@ -91,136 +91,136 @@ public class LanceDbTableLsm {
|
||||
}
|
||||
|
||||
/**
|
||||
* Install an {@link LsmWriteSpec} on this table, selecting the MemWAL LSM write path for future
|
||||
* {@code mergeInsert} calls.
|
||||
* Install an {@link MemWalWriteSpec} on this table, selecting the MemWAL write path for
|
||||
* future {@code mergeInsert} calls.
|
||||
*
|
||||
* <p>All variants require the table to have an unenforced primary key; bucket sharding
|
||||
* additionally requires it to be the single column being bucketed.
|
||||
*/
|
||||
public void setLsmWriteSpec(LsmWriteSpec spec) {
|
||||
public void enableMemWal(MemWalWriteSpec spec) {
|
||||
if (spec == null) {
|
||||
throw new IllegalArgumentException("Spec cannot be null");
|
||||
}
|
||||
client.post(route("set_lsm_write_spec"), spec.toRequestBody());
|
||||
client.post(route("enable_memwal"), spec.toRequestBody());
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove the {@link LsmWriteSpec} from this table, reverting to the standard {@code mergeInsert}
|
||||
* write path.
|
||||
* Remove the {@link MemWalWriteSpec} from this table, reverting to the standard {@code
|
||||
* mergeInsert} write path.
|
||||
*
|
||||
* <p>Errors if no spec is currently set.
|
||||
*/
|
||||
public void unsetLsmWriteSpec() {
|
||||
client.post(route("unset_lsm_write_spec"), null);
|
||||
public void disableMemWal() {
|
||||
client.post(route("disable_memwal"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the {@link LsmWriteSpec} currently installed on this table.
|
||||
* Read the {@link MemWalWriteSpec} currently installed on this table.
|
||||
*
|
||||
* <p>Empty when the LSM write path is not enabled at all — no spec has been set, or one was
|
||||
* removed with {@link #unsetLsmWriteSpec()}. That is a different answer from a present spec
|
||||
* whose {@link LsmWriteSpec#maintainedIndexes()} is null, which selects indexes automatically.
|
||||
* <p>Empty when the MemWAL write path is not enabled at all — no spec has been set, or one was
|
||||
* removed with {@link #disableMemWal()}. That is a different answer from a present spec whose
|
||||
* {@link MemWalWriteSpec#maintainedIndexes()} is null, which selects indexes automatically.
|
||||
*
|
||||
* <p>The spec read back is the one that was installed, selection included: null maintains every
|
||||
* supported index the table has now or gains later, an empty list maintains none, and a
|
||||
* non-empty list maintains exactly those. All three round-trip.
|
||||
* supported index the table has now or gains later, an empty list maintains none, and a non-empty
|
||||
* list maintains exactly those. All three round-trip.
|
||||
*/
|
||||
public Optional<LsmWriteSpec> getLsmWriteSpec() {
|
||||
JsonNode response = client.post(route("get_lsm_write_spec"), null);
|
||||
if (response == null || !response.hasNonNull("lsm_write_spec")) {
|
||||
public Optional<MemWalWriteSpec> getMemWalSpec() {
|
||||
JsonNode response = client.post(route("get_memwal_spec"), null);
|
||||
if (response == null || !response.hasNonNull("memwal_spec")) {
|
||||
return Optional.empty();
|
||||
}
|
||||
return Optional.of(LsmWriteSpec.fromJson(response.get("lsm_write_spec")));
|
||||
return Optional.of(MemWalWriteSpec.fromJson(response.get("memwal_spec")));
|
||||
}
|
||||
|
||||
/**
|
||||
* Freeze every table shard's active memtable into a new SSTable.
|
||||
*
|
||||
* <p>Returns once the freeze is committed. Freezing an empty memtable is a no-op, so this is safe to
|
||||
* call repeatedly.
|
||||
* <p>Returns once the freeze is committed. Freezing an empty memtable is a no-op, so this is safe
|
||||
* to call repeatedly.
|
||||
*/
|
||||
public void flushLsm() {
|
||||
client.post(route("flush_lsm"), null);
|
||||
public void flushMemWal() {
|
||||
client.post(route("flush_memwal"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger a background SSTable compaction pass per table shard.
|
||||
*
|
||||
* <p>Returns once the passes are <em>dispatched</em>, not once they finish — watch {@link
|
||||
* #getLsmStats}, or use {@link #checkpointLsm} to wait for convergence.
|
||||
* #getMemWalStats}, or use {@link #checkpointMemWal} to wait for convergence.
|
||||
*/
|
||||
public void compactLsm() {
|
||||
client.post(route("compact_lsm"), null);
|
||||
public void compactMemWal() {
|
||||
client.post(route("compact_memwal"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read live per-table-shard LSM state.
|
||||
* Read live per-table-shard MemWAL state.
|
||||
*
|
||||
* <p>Answers "how far behind is my fresh tier", "which table shard is hot", and "why is my fresh-tier
|
||||
* vector search brute-force". Mutates no table state.
|
||||
* <p>Answers "how far behind is my fresh tier", "which table shard is hot", and "why is my
|
||||
* fresh-tier vector search brute-force". Mutates no table state.
|
||||
*
|
||||
* <p>Empty only when the LSM write path is not enabled — that is, when the server sends an absent
|
||||
* or null {@code lsm_stats}. A stats object that is present is decoded strictly, and a malformed
|
||||
* one throws rather than decoding to something empty, because {@link #checkpointLsm} reads
|
||||
* convergence out of these numbers and cannot tell a defaulted array from a drained one.
|
||||
* <p>Empty only when the MemWAL write path is not enabled — that is, when the server sends an
|
||||
* absent or null {@code memwal_stats}. A stats object that is present is decoded strictly, and a
|
||||
* malformed one throws rather than decoding to something empty, because {@link #checkpointMemWal}
|
||||
* reads convergence out of these numbers and cannot tell a defaulted array from a drained one.
|
||||
*
|
||||
* @param includeSstableRows Also count rows per SSTable. Off by default because each
|
||||
* count opens an uncached Lance dataset.
|
||||
* @param includeSstableRows Also count rows per SSTable. Off by default because each count opens
|
||||
* an uncached Lance dataset.
|
||||
* @throws IllegalStateException if the response is absent or does not decode.
|
||||
*/
|
||||
public Optional<LsmStats> getLsmStats(boolean includeSstableRows) {
|
||||
public Optional<MemWalStats> getMemWalStats(boolean includeSstableRows) {
|
||||
Map<String, Object> body = new LinkedHashMap<String, Object>();
|
||||
body.put("include_sstable_rows", includeSstableRows);
|
||||
JsonNode response = client.post(route("get_lsm_stats"), body);
|
||||
JsonNode response = client.post(route("get_memwal_stats"), body);
|
||||
if (response == null) {
|
||||
throw new IllegalStateException("get_lsm_stats returned an empty response body");
|
||||
throw new IllegalStateException("get_memwal_stats returned an empty response body");
|
||||
}
|
||||
JsonNode stats = response.get("lsm_stats");
|
||||
JsonNode stats = response.get("memwal_stats");
|
||||
if (stats == null || stats.isNull()) {
|
||||
return Optional.empty();
|
||||
}
|
||||
return Optional.of(LsmStats.fromJson(stats));
|
||||
return Optional.of(MemWalStats.fromJson(stats));
|
||||
}
|
||||
|
||||
/** Equivalent to {@code getLsmStats(false)}. */
|
||||
public Optional<LsmStats> getLsmStats() {
|
||||
return getLsmStats(false);
|
||||
/** Equivalent to {@code getMemWalStats(false)}. */
|
||||
public Optional<MemWalStats> getMemWalStats() {
|
||||
return getMemWalStats(false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Converge this table's LSM write path into its base table.
|
||||
* Converge this table's MemWAL write path into its base table.
|
||||
*
|
||||
* <p>Freezes once, fixes a target watermark from the resulting SSTables, then triggers compaction and
|
||||
* polls until those SSTables are gone. The target set is fixed at the start, so sstables created
|
||||
* <em>during</em> the checkpoint are ignored — that is what lets it terminate under write load,
|
||||
* and what makes it best-effort: it converges the fresh tier as of some instant. Idempotent,
|
||||
* abandonable at any point, safe on a cadence.
|
||||
* <p>Freezes once, fixes a target watermark from the resulting SSTables, then triggers compaction
|
||||
* and polls until those SSTables are gone. The target set is fixed at the start, so sstables
|
||||
* created <em>during</em> the checkpoint are ignored — that is what lets it terminate under write
|
||||
* load, and what makes it best-effort: it converges the fresh tier as of some instant.
|
||||
* Idempotent, abandonable at any point, safe on a cadence.
|
||||
*
|
||||
* <p>The loop runs here, not on the server: {@link #compactLsm} dispatches a pass and returns, so
|
||||
* nothing holds a socket and a client can vanish mid-operation with nothing to reconcile.
|
||||
* <p>The loop runs here, not on the server: {@link #compactMemWal} dispatches a pass and returns,
|
||||
* so nothing holds a socket and a client can vanish mid-operation with nothing to reconcile.
|
||||
* Completion is read from generation numbers in the shard manifest — durable state, unlike a
|
||||
* count in a compact response, which a concurrent write invalidates.
|
||||
*
|
||||
* <p>No liveness bound — the caller owns the deadline. The compactor pool is shared across
|
||||
* tables, so a checkpoint queued behind unrelated work looks exactly like one that is merging.
|
||||
*/
|
||||
public void checkpointLsm() {
|
||||
public void checkpointMemWal() {
|
||||
for (int reissue = 0; reissue <= MAX_REISSUES; reissue++) {
|
||||
// The freeze turns everything written before this call into an SSTable, so the
|
||||
// watermark has to be read after it. Idempotent: freezing an empty memtable is a
|
||||
// no-op, so a re-issue does not churn empty sstables.
|
||||
if (issueVoid(this::flushLsm)) {
|
||||
if (issueVoid(this::flushMemWal)) {
|
||||
backoff(reissue);
|
||||
continue;
|
||||
}
|
||||
|
||||
Attempt<Optional<LsmStats>> stats = issue(() -> getLsmStats(false));
|
||||
Attempt<Optional<MemWalStats>> stats = issue(() -> getMemWalStats(false));
|
||||
if (stats.lostClaim) {
|
||||
backoff(reissue);
|
||||
continue;
|
||||
}
|
||||
if (!stats.value.isPresent()) {
|
||||
// Not WAL-backed; flushLsm would have errored first but for a race.
|
||||
// Not WAL-backed; flushMemWal would have errored first but for a race.
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -235,7 +235,7 @@ public class LanceDbTableLsm {
|
||||
backoff(reissue);
|
||||
}
|
||||
throw new IllegalStateException(
|
||||
"checkpointLsm: the owning node kept losing its claim; re-issued from flush the maximum "
|
||||
"checkpointMemWal: the owning node kept losing its claim; re-issued from flush the maximum "
|
||||
+ "number of times");
|
||||
}
|
||||
|
||||
@@ -246,7 +246,7 @@ public class LanceDbTableLsm {
|
||||
*/
|
||||
private boolean drainToTargets(Map<String, Long> targets) {
|
||||
while (true) {
|
||||
Attempt<Optional<LsmStats>> stats = issue(() -> getLsmStats(false));
|
||||
Attempt<Optional<MemWalStats>> stats = issue(() -> getMemWalStats(false));
|
||||
if (stats.lostClaim) {
|
||||
return false;
|
||||
}
|
||||
@@ -277,7 +277,7 @@ public class LanceDbTableLsm {
|
||||
|
||||
if (!allCompacting) {
|
||||
try {
|
||||
compactLsm();
|
||||
compactMemWal();
|
||||
} catch (LanceDbRestClient.HttpException e) {
|
||||
if (isLostClaim(e)) {
|
||||
return false;
|
||||
@@ -296,7 +296,7 @@ public class LanceDbTableLsm {
|
||||
}
|
||||
|
||||
/** The newest SSTable generation held by each table shard, skipping table shards holding none. */
|
||||
private static Map<String, Long> newestSstableGenerations(LsmStats stats) {
|
||||
private static Map<String, Long> newestSstableGenerations(MemWalStats stats) {
|
||||
Map<String, Long> targets = new HashMap<String, Long>();
|
||||
for (TableShardStats tableShard : stats.tableShards()) {
|
||||
OptionalLong newest = tableShard.newestSstableGeneration();
|
||||
+9
-9
@@ -20,18 +20,18 @@ import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Live per-table-shard LSM state, as returned by {@link LanceDbTableLsm#getLsmStats()}.
|
||||
* Live per-table-shard MemWAL state, as returned by {@link LanceDbTableMemWal#getMemWalStats()}.
|
||||
*
|
||||
* <p>Nothing here is derived: sums and differences (total SSTable bytes, WAL lag) are the caller's to
|
||||
* compute. There is no "LSM is off" shape — that case is an empty {@link java.util.Optional},
|
||||
* <p>Nothing here is derived: sums and differences (total SSTable bytes, WAL lag) are the caller's
|
||||
* to compute. There is no "LSM is off" shape — that case is an empty {@link java.util.Optional},
|
||||
* because a stats object of zeros would read as measurements.
|
||||
*/
|
||||
public class LsmStats {
|
||||
private static final String CONTEXT = "lsm stats";
|
||||
public class MemWalStats {
|
||||
private static final String CONTEXT = "memwal stats";
|
||||
|
||||
private final List<TableShardStats> tableShards;
|
||||
|
||||
LsmStats(List<TableShardStats> tableShards) {
|
||||
MemWalStats(List<TableShardStats> tableShards) {
|
||||
this.tableShards = Collections.unmodifiableList(tableShards);
|
||||
}
|
||||
|
||||
@@ -40,17 +40,17 @@ public class LsmStats {
|
||||
return tableShards;
|
||||
}
|
||||
|
||||
static LsmStats fromJson(JsonNode node) {
|
||||
static MemWalStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
List<TableShardStats> tableShards = new ArrayList<TableShardStats>();
|
||||
for (JsonNode tableShard : JsonFields.requiredArray(node, "table_shards", CONTEXT)) {
|
||||
tableShards.add(TableShardStats.fromJson(tableShard));
|
||||
}
|
||||
return new LsmStats(tableShards);
|
||||
return new MemWalStats(tableShards);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "LsmStats{tableShards=" + tableShards + "}";
|
||||
return "MemWalStats{tableShards=" + tableShards + "}";
|
||||
}
|
||||
}
|
||||
+21
-20
@@ -27,16 +27,16 @@ import java.util.Map;
|
||||
*
|
||||
* <p>Construct via {@link #bucket}, {@link #identity}, or {@link #unsharded}, then optionally chain
|
||||
* {@link #withMaintainedIndexes} and {@link #withWriterConfigDefaults}. Install it with {@link
|
||||
* LanceDbTableLsm#setLsmWriteSpec} and remove it with {@link LanceDbTableLsm#unsetLsmWriteSpec}.
|
||||
* LanceDbTableMemWal#enableMemWal} and remove it with {@link LanceDbTableMemWal#disableMemWal}.
|
||||
*
|
||||
* <p>This is deliberately not {@code org.lance.memwal.InitializeMemWalParams}. That type is Lance's
|
||||
* own, and its maintained-index default is the opposite of this one: it defaults to maintaining
|
||||
* <em>nothing</em>, while a fresh spec here maintains <em>every</em> index. It also cannot express
|
||||
* the null that asks the server to resolve the set.
|
||||
*/
|
||||
public class LsmWriteSpec {
|
||||
public class MemWalWriteSpec {
|
||||
|
||||
private static final String CONTEXT = "get_lsm_write_spec response";
|
||||
private static final String CONTEXT = "get_memwal_spec response";
|
||||
|
||||
/** How writes are routed to MemWAL shards. */
|
||||
public enum Sharding {
|
||||
@@ -73,7 +73,7 @@ public class LsmWriteSpec {
|
||||
private final List<String> maintainedIndexes;
|
||||
private final Map<String, String> writerConfigDefaults;
|
||||
|
||||
private LsmWriteSpec(
|
||||
private MemWalWriteSpec(
|
||||
Sharding sharding,
|
||||
String column,
|
||||
Integer numBuckets,
|
||||
@@ -95,11 +95,11 @@ public class LsmWriteSpec {
|
||||
* @param column A non-nested column with a supported scalar type.
|
||||
* @param numBuckets The number of buckets, in {@code [1, 1024]}.
|
||||
*/
|
||||
public static LsmWriteSpec bucket(String column, int numBuckets) {
|
||||
public static MemWalWriteSpec bucket(String column, int numBuckets) {
|
||||
if (column == null || column.trim().isEmpty()) {
|
||||
throw new IllegalArgumentException("Column cannot be null or empty");
|
||||
}
|
||||
return new LsmWriteSpec(
|
||||
return new MemWalWriteSpec(
|
||||
Sharding.BUCKET, column, numBuckets, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
@@ -111,16 +111,17 @@ public class LsmWriteSpec {
|
||||
* with a given primary key must always produce the same {@code column} value, or upserts of that
|
||||
* key can land in different shards and a stale version can win.
|
||||
*/
|
||||
public static LsmWriteSpec identity(String column) {
|
||||
public static MemWalWriteSpec identity(String column) {
|
||||
if (column == null || column.trim().isEmpty()) {
|
||||
throw new IllegalArgumentException("Column cannot be null or empty");
|
||||
}
|
||||
return new LsmWriteSpec(Sharding.IDENTITY, column, null, null, new HashMap<String, String>());
|
||||
return new MemWalWriteSpec(
|
||||
Sharding.IDENTITY, column, null, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
/** No sharding — every write goes to a single MemWAL shard — maintaining every index. */
|
||||
public static LsmWriteSpec unsharded() {
|
||||
return new LsmWriteSpec(Sharding.UNSHARDED, null, null, null, new HashMap<String, String>());
|
||||
public static MemWalWriteSpec unsharded() {
|
||||
return new MemWalWriteSpec(Sharding.UNSHARDED, null, null, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -131,8 +132,8 @@ public class LsmWriteSpec {
|
||||
*
|
||||
* <p>Note that {@code null} and the empty list mean opposite things here.
|
||||
*/
|
||||
public LsmWriteSpec withMaintainedIndexes(List<String> maintainedIndexes) {
|
||||
return new LsmWriteSpec(
|
||||
public MemWalWriteSpec withMaintainedIndexes(List<String> maintainedIndexes) {
|
||||
return new MemWalWriteSpec(
|
||||
sharding,
|
||||
column,
|
||||
numBuckets,
|
||||
@@ -149,11 +150,11 @@ public class LsmWriteSpec {
|
||||
* max_unflushed_memtable_bytes}, and {@code enable_memtable}. Duration knobs carry an {@code _ms}
|
||||
* suffix, such as {@code max_wal_flush_interval_ms}.
|
||||
*/
|
||||
public LsmWriteSpec withWriterConfigDefaults(Map<String, String> writerConfigDefaults) {
|
||||
public MemWalWriteSpec withWriterConfigDefaults(Map<String, String> writerConfigDefaults) {
|
||||
if (writerConfigDefaults == null) {
|
||||
throw new IllegalArgumentException("writerConfigDefaults cannot be null");
|
||||
}
|
||||
return new LsmWriteSpec(
|
||||
return new MemWalWriteSpec(
|
||||
sharding,
|
||||
column,
|
||||
numBuckets,
|
||||
@@ -191,7 +192,7 @@ public class LsmWriteSpec {
|
||||
return Collections.unmodifiableMap(writerConfigDefaults);
|
||||
}
|
||||
|
||||
/** Render this spec as the {@code set_lsm_write_spec} request body. */
|
||||
/** Render this spec as the {@code enable_memwal} request body. */
|
||||
Map<String, Object> toRequestBody() {
|
||||
Map<String, Object> shardingBody = new LinkedHashMap<String, Object>();
|
||||
shardingBody.put("mode", sharding.wireName());
|
||||
@@ -211,15 +212,15 @@ public class LsmWriteSpec {
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild a spec from a {@code get_lsm_write_spec} response body.
|
||||
* Rebuild a spec from a {@code get_memwal_spec} response body.
|
||||
*
|
||||
* <p>A null {@code maintained_indexes} selects every index the table has; an empty list selects
|
||||
* none.
|
||||
*/
|
||||
static LsmWriteSpec fromJson(JsonNode node) {
|
||||
static MemWalWriteSpec fromJson(JsonNode node) {
|
||||
JsonNode shardingNode = node.get("sharding");
|
||||
if (shardingNode == null || shardingNode.get("mode") == null) {
|
||||
throw new IllegalStateException("get_lsm_write_spec response has no sharding mode");
|
||||
throw new IllegalStateException("get_memwal_spec response has no sharding mode");
|
||||
}
|
||||
Sharding sharding = Sharding.fromWireName(shardingNode.get("mode").asText());
|
||||
|
||||
@@ -252,12 +253,12 @@ public class LsmWriteSpec {
|
||||
.forEachRemaining(name -> defaults.put(name, defaultsNode.get(name).asText()));
|
||||
}
|
||||
|
||||
return new LsmWriteSpec(sharding, column, numBuckets, maintainedIndexes, defaults);
|
||||
return new MemWalWriteSpec(sharding, column, numBuckets, maintainedIndexes, defaults);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "LsmWriteSpec{sharding="
|
||||
return "MemWalWriteSpec{sharding="
|
||||
+ sharding
|
||||
+ ", column="
|
||||
+ column
|
||||
@@ -22,8 +22,8 @@ import java.util.Optional;
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/**
|
||||
* Live state of one table shard. A table is N table shards on one node; flattening to a single number hides
|
||||
* the one hot table shard that is usually why someone opened this endpoint.
|
||||
* Live state of one table shard. A table is N table shards on one node; flattening to a single
|
||||
* number hides the one hot table shard that is usually why someone opened this endpoint.
|
||||
*/
|
||||
public class TableShardStats {
|
||||
private static final String CONTEXT = "tableShard stats";
|
||||
@@ -115,7 +115,9 @@ public class TableShardStats {
|
||||
return compacting;
|
||||
}
|
||||
|
||||
/** Oldest first, active last. Empty for a {@code "Sealed"} table shard, whose state is torn down. */
|
||||
/**
|
||||
* Oldest first, active last. Empty for a {@code "Sealed"} table shard, whose state is torn down.
|
||||
*/
|
||||
public Optional<List<MemtableStats>> memtables() {
|
||||
return Optional.ofNullable(memtables);
|
||||
}
|
||||
|
||||
+136
-133
@@ -46,12 +46,12 @@ import static org.junit.jupiter.api.Assertions.*;
|
||||
* <p>The wire assertions mirror the Rust mocked-endpoint tests in {@code
|
||||
* rust/lancedb/src/remote/table.rs}, which are the contract these routes have to match.
|
||||
*/
|
||||
public class LanceDbTableLsmTest {
|
||||
public class LanceDbTableMemWalTest {
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
|
||||
private HttpServer server;
|
||||
private LanceDbRestClient client;
|
||||
private LanceDbTableLsm lsm;
|
||||
private LanceDbTableMemWal memWal;
|
||||
|
||||
private final List<String> requestPaths = Collections.synchronizedList(new ArrayList<String>());
|
||||
private final List<String> requestBodies = Collections.synchronizedList(new ArrayList<String>());
|
||||
@@ -101,7 +101,7 @@ public class LanceDbTableLsmTest {
|
||||
.database("test-db")
|
||||
.endpoint("http://127.0.0.1:" + server.getAddress().getPort())
|
||||
.buildRestClient();
|
||||
lsm = new LanceDbTableLsm(client, "my_table");
|
||||
memWal = new LanceDbTableMemWal(client, "my_table");
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
@@ -115,12 +115,12 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecUnsharded() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalUnsharded() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded());
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded());
|
||||
|
||||
assertEquals("/v1/table/my_table/set_lsm_write_spec/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/enable_memwal/", requestPaths.get(0));
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("unsharded", body.get("sharding").get("mode").asText());
|
||||
assertFalse(body.get("sharding").has("column"));
|
||||
@@ -128,14 +128,14 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecBucket() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalBucket() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(
|
||||
LsmWriteSpec.tableShard("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
|
||||
memWal.enableMemWal(
|
||||
MemWalWriteSpec.bucket("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("tableShard", body.get("sharding").get("mode").asText());
|
||||
assertEquals("bucket", body.get("sharding").get("mode").asText());
|
||||
assertEquals("id", body.get("sharding").get("column").asText());
|
||||
assertEquals(16, body.get("sharding").get("num_buckets").asInt());
|
||||
assertEquals(1, body.get("maintained_indexes").size());
|
||||
@@ -143,10 +143,10 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecIdentity() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalIdentity() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.identity("tenant"));
|
||||
memWal.enableMemWal(MemWalWriteSpec.identity("tenant"));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("identity", body.get("sharding").get("mode").asText());
|
||||
@@ -160,55 +160,55 @@ public class LanceDbTableLsmTest {
|
||||
*/
|
||||
@Test
|
||||
public void testMaintainedIndexesNullAndEmptyAreDistinctOnTheWire() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded());
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded());
|
||||
JsonNode fresh = MAPPER.readTree(requestBodies.get(0));
|
||||
assertTrue(fresh.has("maintained_indexes"), "the key must be present");
|
||||
assertTrue(fresh.get("maintained_indexes").isNull(), "a fresh spec sends null, not []");
|
||||
|
||||
lsm.setLsmWriteSpec(
|
||||
LsmWriteSpec.unsharded().withMaintainedIndexes(Collections.<String>emptyList()));
|
||||
memWal.enableMemWal(
|
||||
MemWalWriteSpec.unsharded().withMaintainedIndexes(Collections.<String>emptyList()));
|
||||
JsonNode none = MAPPER.readTree(requestBodies.get(1));
|
||||
assertTrue(none.get("maintained_indexes").isArray());
|
||||
assertEquals(0, none.get("maintained_indexes").size());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecWriterConfigDefaults() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalWriterConfigDefaults() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
Map<String, String> defaults = new HashMap<String, String>();
|
||||
defaults.put("max_memtable_rows", "50000");
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded().withWriterConfigDefaults(defaults));
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded().withWriterConfigDefaults(defaults));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("50000", body.get("writer_config_defaults").get("max_memtable_rows").asText());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testUnsetLsmWriteSpec() {
|
||||
enqueue("unset_lsm_write_spec", 200, "");
|
||||
public void testDisableMemWal() {
|
||||
enqueue("disable_memwal", 200, "");
|
||||
|
||||
lsm.unsetLsmWriteSpec();
|
||||
memWal.disableMemWal();
|
||||
|
||||
assertEquals("/v1/table/my_table/unset_lsm_write_spec/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/disable_memwal/", requestPaths.get(0));
|
||||
assertEquals("", requestBodies.get(0));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpec() {
|
||||
public void testGetMemWalSpec() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"tableShard\",\"column\":\"id\","
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"bucket\",\"column\":\"id\","
|
||||
+ "\"num_buckets\":16},\"maintained_indexes\":[\"id_idx\"],"
|
||||
+ "\"writer_config_defaults\":{\"durable_write\":\"true\"}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertEquals(LsmWriteSpec.Sharding.BUCKET, spec.get().sharding());
|
||||
assertEquals(MemWalWriteSpec.Sharding.BUCKET, spec.get().sharding());
|
||||
assertEquals("id", spec.get().column());
|
||||
assertEquals(Integer.valueOf(16), spec.get().numBuckets());
|
||||
assertEquals(Arrays.asList("id_idx"), spec.get().maintainedIndexes());
|
||||
@@ -217,28 +217,28 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
/** A null selection, every index the table has, reads back as null rather than an empty list. */
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRoundTripsEveryIndexSelection() {
|
||||
public void testGetMemWalSpecRoundTripsEveryIndexSelection() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":null,\"writer_config_defaults\":{}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertNull(spec.get().maintainedIndexes());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRoundTripsNoIndexSelection() {
|
||||
public void testGetMemWalSpecRoundTripsNoIndexSelection() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":[],\"writer_config_defaults\":{}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertEquals(Collections.emptyList(), spec.get().maintainedIndexes());
|
||||
@@ -249,32 +249,32 @@ public class LanceDbTableLsmTest {
|
||||
* rejected: reading a string or an object as null would turn a malformed response into a policy.
|
||||
*/
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRejectsANonArrayMaintainedIndexes() {
|
||||
public void testGetMemWalSpecRejectsANonArrayMaintainedIndexes() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":\"id_idx\",\"writer_config_defaults\":{}}}");
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> lsm.getLsmWriteSpec());
|
||||
assertThrows(IllegalStateException.class, () -> memWal.getMemWalSpec());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRejectsNonStringIndexNames() {
|
||||
public void testGetMemWalSpecRejectsNonStringIndexNames() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":[7],\"writer_config_defaults\":{}}}");
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> lsm.getLsmWriteSpec());
|
||||
assertThrows(IllegalStateException.class, () -> memWal.getMemWalSpec());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecAbsent() {
|
||||
enqueue("get_lsm_write_spec", 200, "{\"lsm_write_spec\":null}");
|
||||
public void testGetMemWalSpecAbsent() {
|
||||
enqueue("get_memwal_spec", 200, "{\"memwal_spec\":null}");
|
||||
|
||||
assertFalse(lsm.getLsmWriteSpec().isPresent());
|
||||
assertFalse(memWal.getMemWalSpec().isPresent());
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -282,12 +282,12 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testGetLsmStats() throws Exception {
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
public void testGetMemWalStats() throws Exception {
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
|
||||
Optional<LsmStats> got = lsm.getLsmStats(true);
|
||||
Optional<MemWalStats> got = memWal.getMemWalStats(true);
|
||||
|
||||
assertEquals("/v1/table/my_table/get_lsm_stats/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/get_memwal_stats/", requestPaths.get(0));
|
||||
assertTrue(MAPPER.readTree(requestBodies.get(0)).get("include_sstable_rows").asBoolean());
|
||||
assertTrue(got.isPresent());
|
||||
TableShardStats decoded = got.get().tableShards().get(0);
|
||||
@@ -305,18 +305,18 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
/** The optional fields decode when the server does send them. */
|
||||
@Test
|
||||
public void testGetLsmStatsDecodesOptionalFields() {
|
||||
public void testGetMemWalStatsDecodesOptionalFields() {
|
||||
enqueue(
|
||||
"get_lsm_stats",
|
||||
"get_memwal_stats",
|
||||
200,
|
||||
"{\"lsm_stats\":{\"tableShards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":3,\"wal_entry_position_last_seen\":11,"
|
||||
+ "\"sstables\":[{\"generation\":7,\"bytes\":1024,\"rows\":42}],"
|
||||
+ "\"compacting\":true,\"memtables\":[{\"generation\":8,\"rows\":5,"
|
||||
+ "\"bytes\":64,\"batches\":2,\"indexes\":[\"id_idx\"]}]}]}}");
|
||||
|
||||
TableShardStats decoded = lsm.getLsmStats(true).get().tableShards().get(0);
|
||||
TableShardStats decoded = memWal.getMemWalStats(true).get().tableShards().get(0);
|
||||
|
||||
assertEquals(3, decoded.replayAfterWalEntryPosition());
|
||||
assertEquals(11, decoded.walEntryPositionLastSeen());
|
||||
@@ -332,17 +332,17 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmStatsAbsentWhenLsmDisabled() {
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
public void testGetMemWalStatsAbsentWhenMemWalDisabled() {
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
assertFalse(lsm.getLsmStats().isPresent());
|
||||
assertFalse(memWal.getMemWalStats().isPresent());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmStatsDefaultsToExcludingGenerationRows() throws Exception {
|
||||
enqueue("get_lsm_stats", 200, stats());
|
||||
public void testGetMemWalStatsDefaultsToExcludingGenerationRows() throws Exception {
|
||||
enqueue("get_memwal_stats", 200, stats());
|
||||
|
||||
lsm.getLsmStats();
|
||||
memWal.getMemWalStats();
|
||||
|
||||
assertFalse(MAPPER.readTree(requestBodies.get(0)).get("include_sstable_rows").asBoolean());
|
||||
}
|
||||
@@ -353,22 +353,22 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
@Test
|
||||
public void testFlushAndCompactRoutes() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("compact_lsm", 200, "");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("compact_memwal", 200, "");
|
||||
|
||||
lsm.flushLsm();
|
||||
lsm.compactLsm();
|
||||
memWal.flushMemWal();
|
||||
memWal.compactMemWal();
|
||||
|
||||
assertEquals("/v1/table/my_table/flush_lsm/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/compact_lsm/", requestPaths.get(1));
|
||||
assertEquals("/v1/table/my_table/flush_memwal/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/compact_memwal/", requestPaths.get(1));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testHttpErrorCarriesStatus() {
|
||||
enqueue("flush_lsm", 404, "no such table");
|
||||
enqueue("flush_memwal", 404, "no such table");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.flushLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.flushMemWal());
|
||||
assertEquals(404, e.statusCode());
|
||||
}
|
||||
|
||||
@@ -377,97 +377,98 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testCheckpointReturnsWhenLsmDisabled() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
public void testCheckpointReturnsWhenMemWalDisabled() {
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"), "nothing to compact when the LSM path is off");
|
||||
assertEquals(0, countCalls("compact_memwal"), "nothing to compact when the LSM path is off");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointReturnsWhenNoGenerationsOutstanding() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
// A table shard with no SSTables yields no target, so the drain never starts.
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"));
|
||||
assertEquals(0, countCalls("compact_memwal"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointConvergesOnceTargetGenerationsAreGone() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
// Watermark read: shard-0 holds sstables 7 and 8, so target = 8.
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
// First drain poll: both still outstanding, nothing compacting -> dispatch a pass.
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
// Second drain poll: drained past the target -> done.
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 9L)));
|
||||
enqueue("compact_lsm", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 9L)));
|
||||
enqueue("compact_memwal", 200, "");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(1, countCalls("compact_lsm"), "one pass dispatched");
|
||||
assertEquals(3, countCalls("get_lsm_stats"), "watermark read plus two drain polls");
|
||||
assertEquals(1, countCalls("compact_memwal"), "one pass dispatched");
|
||||
assertEquals(3, countCalls("get_memwal_stats"), "watermark read plus two drain polls");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointDoesNotPileOnWhileEveryTargetBucketIsCompacting() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", true, 4L)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", true, 4L)));
|
||||
// Still compacting on the first poll, so no pass is dispatched; then it drains.
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", true, 4L)));
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false, 5L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", true, 4L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 5L)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"), "a latched tableShard is left alone");
|
||||
assertEquals(0, countCalls("compact_memwal"), "a latched table shard is left alone");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointRetriesFromFlushAfterLostClaim() {
|
||||
// 421 on the watermark read: the node lost its claim, so the whole thing restarts
|
||||
// from flush rather than retrying the read in place.
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 421, "no claim");
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 421, "no claim");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(2, countCalls("flush_lsm"), "re-issued from flush");
|
||||
assertEquals(2, countCalls("flush_memwal"), "re-issued from flush");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointRetriesRetryableStatusInPlace() {
|
||||
enqueue("flush_lsm", 429, "latch held");
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, stats(tableShard("shard-0", false)));
|
||||
enqueue("flush_memwal", 429, "latch held");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(2, countCalls("flush_lsm"), "429 retried in place, not re-issued");
|
||||
assertEquals(2, countCalls("flush_memwal"), "429 retried in place, not re-issued");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointPropagatesTerminalStatus() {
|
||||
enqueue("flush_lsm", 400, "bad request");
|
||||
enqueue("flush_memwal", 400, "bad request");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.checkpointLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.checkpointMemWal());
|
||||
assertEquals(400, e.statusCode());
|
||||
assertEquals(1, countCalls("flush_lsm"), "a terminal status is not retried");
|
||||
assertEquals(1, countCalls("flush_memwal"), "a terminal status is not retried");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointGivesUpAfterRepeatedLostClaims() {
|
||||
enqueue("flush_lsm", 421, "no claim");
|
||||
enqueue("flush_memwal", 421, "no claim");
|
||||
|
||||
IllegalStateException e = assertThrows(IllegalStateException.class, () -> lsm.checkpointLsm());
|
||||
IllegalStateException e =
|
||||
assertThrows(IllegalStateException.class, () -> memWal.checkpointMemWal());
|
||||
assertTrue(e.getMessage().contains("kept losing its claim"), e.getMessage());
|
||||
assertEquals(4, countCalls("flush_lsm"), "the initial attempt plus MAX_REISSUES");
|
||||
assertEquals(4, countCalls("flush_memwal"), "the initial attempt plus MAX_REISSUES");
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -476,24 +477,25 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
/**
|
||||
* A stats payload that does not decode must fail closed. Every one of these bodies used to be
|
||||
* read as "no tableShards", which is indistinguishable from a drained table, so {@code checkpointLsm}
|
||||
* reported convergence for a checkpoint that never ran.
|
||||
* read as "no table shards", which is indistinguishable from a drained table, so {@code
|
||||
* checkpointMemWal} reported convergence for a checkpoint that never ran.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckpointRejectsMalformedStats() {
|
||||
Map<String, String> malformed = new LinkedHashMap<String, String>();
|
||||
malformed.put("no response body at all", "");
|
||||
malformed.put("stats object with no tableShards", "{\"lsm_stats\":{}}");
|
||||
malformed.put("tableShard missing its required fields", "{\"lsm_stats\":{\"tableShards\":[{}]}}");
|
||||
malformed.put("stats object with no table shards", "{\"memwal_stats\":{}}");
|
||||
malformed.put(
|
||||
"tableShard missing sstables",
|
||||
"{\"lsm_stats\":{\"tableShards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"table shard missing its required fields", "{\"memwal_stats\":{\"table_shards\":[{}]}}");
|
||||
malformed.put(
|
||||
"table shard missing sstables",
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
|
||||
+ "\"compacting\":false}]}}");
|
||||
malformed.put(
|
||||
"generation with a non-numeric generation number",
|
||||
"{\"lsm_stats\":{\"tableShards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
|
||||
+ "\"sstables\":[{\"generation\":\"7\",\"bytes\":1024}],"
|
||||
@@ -501,25 +503,25 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
for (Map.Entry<String, String> each : malformed.entrySet()) {
|
||||
setUpFresh();
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, each.getValue());
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, each.getValue());
|
||||
|
||||
assertThrows(
|
||||
IllegalStateException.class,
|
||||
() -> lsm.checkpointLsm(),
|
||||
() -> memWal.checkpointMemWal(),
|
||||
each.getKey() + " must not report convergence");
|
||||
}
|
||||
}
|
||||
|
||||
/** The one shape that legitimately means "this table has no LSM write path". */
|
||||
/** The one shape that legitimately means "this table has no MemWAL write path". */
|
||||
@Test
|
||||
public void testCheckpointTreatsNullStatsAsNotWalBacked() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(1, countCalls("get_lsm_stats"));
|
||||
assertEquals(1, countCalls("get_memwal_stats"));
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -529,18 +531,19 @@ public class LanceDbTableLsmTest {
|
||||
/**
|
||||
* The transport must not retry on the checkpoint loop's behalf. Apache HttpClient's default
|
||||
* strategy retries exactly 429 and 503 — the two statuses {@code isRetryable} owns — which
|
||||
* doubled every budget here and also retried {@code compact_lsm} in place, where the loop is
|
||||
* doubled every budget here and also retried {@code compact_memwal} in place, where the loop is
|
||||
* built to fall through to a fresh stats poll instead.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckpointRetryBudgetIsNotDoubledByTheTransport() {
|
||||
enqueue("flush_lsm", 429, "latch held");
|
||||
enqueue("flush_memwal", 429, "latch held");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.checkpointLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.checkpointMemWal());
|
||||
|
||||
assertEquals(429, e.statusCode(), "the exhausted budget propagates the last error as itself");
|
||||
assertEquals(9, countCalls("flush_lsm"), "the initial request plus MAX_RETRIES, and no more");
|
||||
assertEquals(
|
||||
9, countCalls("flush_memwal"), "the initial request plus MAX_RETRIES, and no more");
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -555,9 +558,9 @@ public class LanceDbTableLsmTest {
|
||||
return numbers;
|
||||
}
|
||||
|
||||
/** Build an {@code lsm_stats} response body from tableShard fragments. */
|
||||
/** Build a {@code memwal_stats} response body from table shard fragments. */
|
||||
private static String stats(String... tableShards) {
|
||||
return "{\"lsm_stats\":{\"tableShards\":[" + String.join(",", tableShards) + "]}}";
|
||||
return "{\"memwal_stats\":{\"table_shards\":[" + String.join(",", tableShards) + "]}}";
|
||||
}
|
||||
|
||||
private static String tableShard(String shardId, boolean compacting, Long... sstables) {
|
||||
@@ -597,7 +600,7 @@ public class LanceDbTableLsmTest {
|
||||
return requestPaths.stream().filter(path -> operationOf(path).equals(operation)).count();
|
||||
}
|
||||
|
||||
/** {@code /v1/table/my_table/flush_lsm/} -> {@code flush_lsm}. */
|
||||
/** {@code /v1/table/my_table/flush_memwal/} -> {@code flush_memwal}. */
|
||||
private static String operationOf(String path) {
|
||||
String[] segments = path.split("/");
|
||||
return segments.length == 0 ? "" : segments[segments.length - 1];
|
||||
Reference in New Issue
Block a user