mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-18 01:32:27 +00:00
refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
0968c21f4c
commit
fe34eb3549
+9
-9
@@ -29,7 +29,7 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.build();
|
||||
```
|
||||
|
||||
## MemWAL LSM write path
|
||||
## MemWAL write path
|
||||
|
||||
Most table operations reach LanceDB through the `LanceNamespace` above, which is
|
||||
generated from the Lance Namespace specification. The MemWAL LSM routes are not part
|
||||
@@ -37,27 +37,27 @@ of that specification, so they are issued through a separate client:
|
||||
|
||||
```java
|
||||
import com.lancedb.LanceDbRestClient;
|
||||
import com.lancedb.LanceDbTableLsm;
|
||||
import com.lancedb.LsmWriteSpec;
|
||||
import com.lancedb.LanceDbTableMemWal;
|
||||
import com.lancedb.MemWalWriteSpec;
|
||||
|
||||
LanceDbRestClient client = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.apiKey("your_lancedb_cloud_api_key")
|
||||
.database("your_database_name")
|
||||
.buildRestClient();
|
||||
|
||||
LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");
|
||||
LanceDbTableMemWal memWal = new LanceDbTableMemWal(client, "my_table");
|
||||
|
||||
// Route future merge_insert upserts through the MemWAL, hash-bucketed by `id`.
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));
|
||||
memWal.enableMemWal(MemWalWriteSpec.bucket("id", 16));
|
||||
|
||||
// ... merge_insert traffic ...
|
||||
|
||||
// Converge the fresh tier into the base table.
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
// Inspect live per-bucket state.
|
||||
lsm.getLsmStats().ifPresent(stats -> stats.buckets().forEach(bucket ->
|
||||
System.out.println(bucket.shardId() + ": " + bucket.generations().size() + " L0 generations")));
|
||||
// Inspect live per-table-shard state.
|
||||
memWal.getMemWalStats().ifPresent(stats -> stats.tableShards().forEach(tableShard ->
|
||||
System.out.println(tableShard.shardId() + ": " + tableShard.sstables().size() + " SSTables")));
|
||||
|
||||
client.close();
|
||||
```
|
||||
|
||||
@@ -20,12 +20,12 @@ import com.fasterxml.jackson.databind.JsonNode;
|
||||
*
|
||||
* <p>Every reader fails closed: a missing, null, or wrong-typed field throws rather than
|
||||
* defaulting. That mirrors the serde decoding the Rust client applies to the same payloads in
|
||||
* {@code rust/lancedb/src/table/lsm_stats.rs}, where a required field has no default and a
|
||||
* {@code rust/lancedb/src/table/memwal_stats.rs}, where a required field has no default and a
|
||||
* malformed response is an error rather than a zero.
|
||||
*
|
||||
* <p>The alternative — Jackson's {@code path()}, which yields a missing node that reads as an empty
|
||||
* array or a zero — is unsafe here because {@link LanceDbTableLsm#checkpointLsm()} decides
|
||||
* convergence from these numbers. A defaulted {@code generations} array is indistinguishable from a
|
||||
* array or a zero — is unsafe here because {@link LanceDbTableMemWal#checkpointMemWal()} decides
|
||||
* convergence from these numbers. A defaulted {@code sstables} array is indistinguishable from a
|
||||
* drained one, so a malformed response would report a checkpoint that never happened.
|
||||
*/
|
||||
final class JsonFields {
|
||||
|
||||
@@ -151,8 +151,8 @@ public class LanceDbNamespaceClientBuilder {
|
||||
* Build a {@link LanceDbRestClient} for the same endpoint.
|
||||
*
|
||||
* <p>Needed only for LanceDB routes that the Lance Namespace specification does not cover — the
|
||||
* MemWAL LSM write path, reached through {@link LanceDbTableLsm}. Every other table operation
|
||||
* belongs on the {@link LanceNamespace} from {@link #build()}.
|
||||
* MemWAL write path, reached through {@link LanceDbTableMemWal}. Every other table
|
||||
* operation belongs on the {@link LanceNamespace} from {@link #build()}.
|
||||
*
|
||||
* <p>The returned client owns an HTTP connection pool; close it when you are done with it.
|
||||
*
|
||||
|
||||
@@ -31,9 +31,9 @@ import java.io.UncheckedIOException;
|
||||
* specification does not cover.
|
||||
*
|
||||
* <p>Most table operations reach LanceDB through {@link org.lance.namespace.LanceNamespace}, which
|
||||
* is generated from the namespace spec. A handful of routes — the MemWAL LSM write path in
|
||||
* is generated from the namespace spec. A handful of routes — the MemWAL write path in
|
||||
* particular — are served by the same endpoint but are not part of that spec, so they are issued
|
||||
* directly here. See {@link LanceDbTableLsm}.
|
||||
* directly here. See {@link LanceDbTableMemWal}.
|
||||
*
|
||||
* <p>Obtain one from {@link LanceDbNamespaceClientBuilder#buildRestClient()}.
|
||||
*/
|
||||
@@ -50,8 +50,8 @@ public class LanceDbRestClient implements Closeable {
|
||||
this.apiKey = apiKey;
|
||||
this.database = database;
|
||||
// Automatic retries off, deliberately. The default strategy retries 429 and 503 —
|
||||
// exactly the two statuses LanceDbTableLsm.checkpointLsm() acts on — which would
|
||||
// silently double its explicit retry budget and would also retry compact_lsm in
|
||||
// exactly the two statuses LanceDbTableMemWal.checkpointMemWal() acts on — which would
|
||||
// silently double its explicit retry budget and would also retry compact_memwal in
|
||||
// place, where the loop is designed to fall through to a fresh stats poll instead.
|
||||
// The checkpoint loop owns the 421/429/503 transitions; the transport must not.
|
||||
this.http = HttpClients.custom().disableAutomaticRetries().build();
|
||||
@@ -98,8 +98,9 @@ public class LanceDbRestClient implements Closeable {
|
||||
/**
|
||||
* A non-2xx response.
|
||||
*
|
||||
* <p>The status is exposed because callers act on it: {@link LanceDbTableLsm#checkpointLsm()}
|
||||
* treats 429 and 503 as retryable and 421 as a lost node claim.
|
||||
* <p>The status is exposed because callers act on it: {@link
|
||||
* LanceDbTableMemWal#checkpointMemWal()} treats 429 and 503 as retryable and 421 as a lost node
|
||||
* claim.
|
||||
*/
|
||||
public static class HttpException extends RuntimeException {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
+90
-90
@@ -22,11 +22,11 @@ import java.util.Optional;
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/**
|
||||
* The MemWAL LSM write path for one LanceDB Cloud or Enterprise table.
|
||||
* The MemWAL write path for one LanceDB Cloud or Enterprise table.
|
||||
*
|
||||
* <p>Installing an {@link LsmWriteSpec} routes {@code mergeInsert} upserts through Lance's MemWAL —
|
||||
* an LSM-style append — instead of the standard merge path. Rows land in an in-memory memtable,
|
||||
* seal into L0 generations, and are merged into the base table by compaction.
|
||||
* <p>Installing an {@link MemWalWriteSpec} routes {@code mergeInsert} upserts through Lance's
|
||||
* MemWAL — an LSM-style append — instead of the standard merge path. Rows land in an in-memory
|
||||
* memtable, freeze into SSTables, and are merged into the base table by compaction.
|
||||
*
|
||||
* <p>These routes are not part of the Lance Namespace specification, so they are issued directly
|
||||
* rather than through {@link org.lance.namespace.LanceNamespace}.
|
||||
@@ -37,23 +37,23 @@ import java.util.OptionalLong;
|
||||
* .database("your_database_name")
|
||||
* .buildRestClient();
|
||||
*
|
||||
* LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");
|
||||
* lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));
|
||||
* LanceDbTableMemWal memWal = new LanceDbTableMemWal(client, "my_table");
|
||||
* memWal.enableMemWal(MemWalWriteSpec.bucket("id", 16));
|
||||
* // ... merge_insert traffic ...
|
||||
* lsm.checkpointLsm();
|
||||
* memWal.checkpointMemWal();
|
||||
* }</pre>
|
||||
*/
|
||||
public class LanceDbTableLsm {
|
||||
public class LanceDbTableMemWal {
|
||||
|
||||
/**
|
||||
* Interval between {@code get_lsm_stats} polls during a checkpoint. One interval is roughly one
|
||||
* compaction pass, the granularity at which the answer can change.
|
||||
* Interval between {@code get_memwal_stats} polls during a checkpoint. One interval is roughly
|
||||
* one compaction pass, the granularity at which the answer can change.
|
||||
*/
|
||||
private static final long POLL_INTERVAL_MS = 5_000L;
|
||||
|
||||
/**
|
||||
* Cap on re-issues from {@code flushLsm} after a 421, so a crash-looping node cannot turn flush →
|
||||
* compact → 421 → flush into a spin.
|
||||
* Cap on re-issues from {@code flushMemWal} after a 421, so a crash-looping node cannot turn
|
||||
* flush → compact → 421 → flush into a spin.
|
||||
*
|
||||
* <p>Deliberately not shared with {@link #MAX_RETRIES}: a claim that keeps evaporating is a
|
||||
* broken node, while contention is routine and wants a real budget.
|
||||
@@ -79,7 +79,7 @@ public class LanceDbTableLsm {
|
||||
* @param tableIdentifier The table's full identifier, {@code $}-delimited when it sits inside a
|
||||
* namespace, such as {@code analytics$events}.
|
||||
*/
|
||||
public LanceDbTableLsm(LanceDbRestClient client, String tableIdentifier) {
|
||||
public LanceDbTableMemWal(LanceDbRestClient client, String tableIdentifier) {
|
||||
if (client == null) {
|
||||
throw new IllegalArgumentException("Client cannot be null");
|
||||
}
|
||||
@@ -91,140 +91,140 @@ public class LanceDbTableLsm {
|
||||
}
|
||||
|
||||
/**
|
||||
* Install an {@link LsmWriteSpec} on this table, selecting the MemWAL LSM write path for future
|
||||
* {@code mergeInsert} calls.
|
||||
* Install an {@link MemWalWriteSpec} on this table, selecting the MemWAL write path for
|
||||
* future {@code mergeInsert} calls.
|
||||
*
|
||||
* <p>All variants require the table to have an unenforced primary key; bucket sharding
|
||||
* additionally requires it to be the single column being bucketed.
|
||||
*/
|
||||
public void setLsmWriteSpec(LsmWriteSpec spec) {
|
||||
public void enableMemWal(MemWalWriteSpec spec) {
|
||||
if (spec == null) {
|
||||
throw new IllegalArgumentException("Spec cannot be null");
|
||||
}
|
||||
client.post(route("set_lsm_write_spec"), spec.toRequestBody());
|
||||
client.post(route("enable_memwal"), spec.toRequestBody());
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove the {@link LsmWriteSpec} from this table, reverting to the standard {@code mergeInsert}
|
||||
* write path.
|
||||
* Remove the {@link MemWalWriteSpec} from this table, reverting to the standard {@code
|
||||
* mergeInsert} write path.
|
||||
*
|
||||
* <p>Errors if no spec is currently set.
|
||||
*/
|
||||
public void unsetLsmWriteSpec() {
|
||||
client.post(route("unset_lsm_write_spec"), null);
|
||||
public void disableMemWal() {
|
||||
client.post(route("disable_memwal"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the {@link LsmWriteSpec} currently installed on this table.
|
||||
* Read the {@link MemWalWriteSpec} currently installed on this table.
|
||||
*
|
||||
* <p>Empty when the LSM write path is not enabled at all — no spec has been set, or one was
|
||||
* removed with {@link #unsetLsmWriteSpec()}. That is a different answer from a present spec
|
||||
* whose {@link LsmWriteSpec#maintainedIndexes()} is null, which selects indexes automatically.
|
||||
* <p>Empty when the MemWAL write path is not enabled at all — no spec has been set, or one was
|
||||
* removed with {@link #disableMemWal()}. That is a different answer from a present spec whose
|
||||
* {@link MemWalWriteSpec#maintainedIndexes()} is null, which selects indexes automatically.
|
||||
*
|
||||
* <p>The spec read back is the one that was installed, selection included: null maintains every
|
||||
* supported index the table has now or gains later, an empty list maintains none, and a
|
||||
* non-empty list maintains exactly those. All three round-trip.
|
||||
* supported index the table has now or gains later, an empty list maintains none, and a non-empty
|
||||
* list maintains exactly those. All three round-trip.
|
||||
*/
|
||||
public Optional<LsmWriteSpec> getLsmWriteSpec() {
|
||||
JsonNode response = client.post(route("get_lsm_write_spec"), null);
|
||||
if (response == null || !response.hasNonNull("lsm_write_spec")) {
|
||||
public Optional<MemWalWriteSpec> getMemWalSpec() {
|
||||
JsonNode response = client.post(route("get_memwal_spec"), null);
|
||||
if (response == null || !response.hasNonNull("memwal_spec")) {
|
||||
return Optional.empty();
|
||||
}
|
||||
return Optional.of(LsmWriteSpec.fromJson(response.get("lsm_write_spec")));
|
||||
return Optional.of(MemWalWriteSpec.fromJson(response.get("memwal_spec")));
|
||||
}
|
||||
|
||||
/**
|
||||
* Seal every bucket's active memtable into a new L0 generation.
|
||||
* Freeze every table shard's active memtable into a new SSTable.
|
||||
*
|
||||
* <p>Returns once the seal is committed. Sealing an empty memtable is a no-op, so this is safe to
|
||||
* call repeatedly.
|
||||
* <p>Returns once the freeze is committed. Freezing an empty memtable is a no-op, so this is safe
|
||||
* to call repeatedly.
|
||||
*/
|
||||
public void flushLsm() {
|
||||
client.post(route("flush_lsm"), null);
|
||||
public void flushMemWal() {
|
||||
client.post(route("flush_memwal"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger a background L0 → base compaction pass per bucket.
|
||||
* Trigger a background SSTable compaction pass per table shard.
|
||||
*
|
||||
* <p>Returns once the passes are <em>dispatched</em>, not once they finish — watch {@link
|
||||
* #getLsmStats}, or use {@link #checkpointLsm} to wait for convergence.
|
||||
* #getMemWalStats}, or use {@link #checkpointMemWal} to wait for convergence.
|
||||
*/
|
||||
public void compactLsm() {
|
||||
client.post(route("compact_lsm"), null);
|
||||
public void compactMemWal() {
|
||||
client.post(route("compact_memwal"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read live per-bucket LSM state.
|
||||
* Read live per-table-shard MemWAL state.
|
||||
*
|
||||
* <p>Answers "how far behind is my fresh tier", "which bucket is hot", and "why is my fresh-tier
|
||||
* vector search brute-force". Mutates no table state.
|
||||
* <p>Answers "how far behind is my fresh tier", "which table shard is hot", and "why is my
|
||||
* fresh-tier vector search brute-force". Mutates no table state.
|
||||
*
|
||||
* <p>Empty only when the LSM write path is not enabled — that is, when the server sends an absent
|
||||
* or null {@code lsm_stats}. A stats object that is present is decoded strictly, and a malformed
|
||||
* one throws rather than decoding to something empty, because {@link #checkpointLsm} reads
|
||||
* convergence out of these numbers and cannot tell a defaulted array from a drained one.
|
||||
* <p>Empty only when the MemWAL write path is not enabled — that is, when the server sends an
|
||||
* absent or null {@code memwal_stats}. A stats object that is present is decoded strictly, and a
|
||||
* malformed one throws rather than decoding to something empty, because {@link #checkpointMemWal}
|
||||
* reads convergence out of these numbers and cannot tell a defaulted array from a drained one.
|
||||
*
|
||||
* @param includeGenerationRows Also count rows per L0 generation. Off by default because each
|
||||
* count opens an uncached Lance dataset.
|
||||
* @param includeSstableRows Also count rows per SSTable. Off by default because each count opens
|
||||
* an uncached Lance dataset.
|
||||
* @throws IllegalStateException if the response is absent or does not decode.
|
||||
*/
|
||||
public Optional<LsmStats> getLsmStats(boolean includeGenerationRows) {
|
||||
public Optional<MemWalStats> getMemWalStats(boolean includeSstableRows) {
|
||||
Map<String, Object> body = new LinkedHashMap<String, Object>();
|
||||
body.put("include_generation_rows", includeGenerationRows);
|
||||
JsonNode response = client.post(route("get_lsm_stats"), body);
|
||||
body.put("include_sstable_rows", includeSstableRows);
|
||||
JsonNode response = client.post(route("get_memwal_stats"), body);
|
||||
if (response == null) {
|
||||
throw new IllegalStateException("get_lsm_stats returned an empty response body");
|
||||
throw new IllegalStateException("get_memwal_stats returned an empty response body");
|
||||
}
|
||||
JsonNode stats = response.get("lsm_stats");
|
||||
JsonNode stats = response.get("memwal_stats");
|
||||
if (stats == null || stats.isNull()) {
|
||||
return Optional.empty();
|
||||
}
|
||||
return Optional.of(LsmStats.fromJson(stats));
|
||||
return Optional.of(MemWalStats.fromJson(stats));
|
||||
}
|
||||
|
||||
/** Equivalent to {@code getLsmStats(false)}. */
|
||||
public Optional<LsmStats> getLsmStats() {
|
||||
return getLsmStats(false);
|
||||
/** Equivalent to {@code getMemWalStats(false)}. */
|
||||
public Optional<MemWalStats> getMemWalStats() {
|
||||
return getMemWalStats(false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Converge this table's LSM write path into its base table.
|
||||
* Converge this table's MemWAL write path into its base table.
|
||||
*
|
||||
* <p>Seals once, fixes a target watermark from the resulting L0, then triggers compaction and
|
||||
* polls until that L0 is gone. The target set is fixed at the start, so generations created
|
||||
* <em>during</em> the checkpoint are ignored — that is what lets it terminate under write load,
|
||||
* and what makes it best-effort: it converges the fresh tier as of some instant. Idempotent,
|
||||
* abandonable at any point, safe on a cadence.
|
||||
* <p>Freezes once, fixes a target watermark from the resulting SSTables, then triggers compaction
|
||||
* and polls until those SSTables are gone. The target set is fixed at the start, so sstables
|
||||
* created <em>during</em> the checkpoint are ignored — that is what lets it terminate under write
|
||||
* load, and what makes it best-effort: it converges the fresh tier as of some instant.
|
||||
* Idempotent, abandonable at any point, safe on a cadence.
|
||||
*
|
||||
* <p>The loop runs here, not on the server: {@link #compactLsm} dispatches a pass and returns, so
|
||||
* nothing holds a socket and a client can vanish mid-operation with nothing to reconcile.
|
||||
* <p>The loop runs here, not on the server: {@link #compactMemWal} dispatches a pass and returns,
|
||||
* so nothing holds a socket and a client can vanish mid-operation with nothing to reconcile.
|
||||
* Completion is read from generation numbers in the shard manifest — durable state, unlike a
|
||||
* count in a compact response, which a concurrent write invalidates.
|
||||
*
|
||||
* <p>No liveness bound — the caller owns the deadline. The compactor pool is shared across
|
||||
* tables, so a checkpoint queued behind unrelated work looks exactly like one that is merging.
|
||||
*/
|
||||
public void checkpointLsm() {
|
||||
public void checkpointMemWal() {
|
||||
for (int reissue = 0; reissue <= MAX_REISSUES; reissue++) {
|
||||
// The seal turns everything written before this call into a generation, so the
|
||||
// watermark has to be read after it. Idempotent: sealing an empty memtable is a
|
||||
// no-op, so a re-issue does not churn empty generations.
|
||||
if (issueVoid(this::flushLsm)) {
|
||||
// The freeze turns everything written before this call into an SSTable, so the
|
||||
// watermark has to be read after it. Idempotent: freezing an empty memtable is a
|
||||
// no-op, so a re-issue does not churn empty sstables.
|
||||
if (issueVoid(this::flushMemWal)) {
|
||||
backoff(reissue);
|
||||
continue;
|
||||
}
|
||||
|
||||
Attempt<Optional<LsmStats>> stats = issue(() -> getLsmStats(false));
|
||||
Attempt<Optional<MemWalStats>> stats = issue(() -> getMemWalStats(false));
|
||||
if (stats.lostClaim) {
|
||||
backoff(reissue);
|
||||
continue;
|
||||
}
|
||||
if (!stats.value.isPresent()) {
|
||||
// Not WAL-backed; flushLsm would have errored first but for a race.
|
||||
// Not WAL-backed; flushMemWal would have errored first but for a race.
|
||||
return;
|
||||
}
|
||||
|
||||
Map<String, Long> targets = newestGenerations(stats.value.get());
|
||||
Map<String, Long> targets = newestSstableGenerations(stats.value.get());
|
||||
if (targets.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
@@ -235,18 +235,18 @@ public class LanceDbTableLsm {
|
||||
backoff(reissue);
|
||||
}
|
||||
throw new IllegalStateException(
|
||||
"checkpointLsm: the owning node kept losing its claim; re-issued from flush the maximum "
|
||||
"checkpointMemWal: the owning node kept losing its claim; re-issued from flush the maximum "
|
||||
+ "number of times");
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger and poll until no bucket holds a generation at or below its target.
|
||||
* Trigger and poll until no table shard holds an SSTable generation at or below its target.
|
||||
*
|
||||
* @return true when the drain finished, false when the table needs re-claiming from flush.
|
||||
*/
|
||||
private boolean drainToTargets(Map<String, Long> targets) {
|
||||
while (true) {
|
||||
Attempt<Optional<LsmStats>> stats = issue(() -> getLsmStats(false));
|
||||
Attempt<Optional<MemWalStats>> stats = issue(() -> getMemWalStats(false));
|
||||
if (stats.lostClaim) {
|
||||
return false;
|
||||
}
|
||||
@@ -254,21 +254,21 @@ public class LanceDbTableLsm {
|
||||
return true;
|
||||
}
|
||||
|
||||
// `compacting` is the bucket's compaction latch, held from dispatch until the pass
|
||||
// `compacting` is the table shard's compaction latch, held from dispatch until the pass
|
||||
// ends — including while it waits on a pod-wide permit. So it answers one question
|
||||
// only: do not pile on. Buckets with nothing outstanding are skipped, not counted
|
||||
// only: do not pile on. Table shards with nothing outstanding are skipped, not counted
|
||||
// as idle.
|
||||
long outstanding = 0;
|
||||
boolean allCompacting = true;
|
||||
for (BucketStats bucket : stats.value.get().buckets()) {
|
||||
Long target = targets.get(bucket.shardId());
|
||||
for (TableShardStats tableShard : stats.value.get().tableShards()) {
|
||||
Long target = targets.get(tableShard.shardId());
|
||||
if (target == null) {
|
||||
continue;
|
||||
}
|
||||
long remaining = bucket.outstandingGenerations(target);
|
||||
long remaining = tableShard.outstandingSstables(target);
|
||||
if (remaining > 0) {
|
||||
outstanding += remaining;
|
||||
allCompacting &= bucket.compacting();
|
||||
allCompacting &= tableShard.compacting();
|
||||
}
|
||||
}
|
||||
if (outstanding == 0) {
|
||||
@@ -277,7 +277,7 @@ public class LanceDbTableLsm {
|
||||
|
||||
if (!allCompacting) {
|
||||
try {
|
||||
compactLsm();
|
||||
compactMemWal();
|
||||
} catch (LanceDbRestClient.HttpException e) {
|
||||
if (isLostClaim(e)) {
|
||||
return false;
|
||||
@@ -285,7 +285,7 @@ public class LanceDbTableLsm {
|
||||
if (!isRetryable(e)) {
|
||||
throw e;
|
||||
}
|
||||
// A 429 here means the server could latch no bucket at all, which the poll
|
||||
// A 429 here means the server could latch no table shard at all, which the poll
|
||||
// above already handles. Not retried in place: the latch it would contend for
|
||||
// is the one doing the work, so fall through and re-read — POLL_INTERVAL_MS is
|
||||
// the backoff.
|
||||
@@ -295,13 +295,13 @@ public class LanceDbTableLsm {
|
||||
}
|
||||
}
|
||||
|
||||
/** The newest generation held by each bucket, skipping buckets holding none. */
|
||||
private static Map<String, Long> newestGenerations(LsmStats stats) {
|
||||
/** The newest SSTable generation held by each table shard, skipping table shards holding none. */
|
||||
private static Map<String, Long> newestSstableGenerations(MemWalStats stats) {
|
||||
Map<String, Long> targets = new HashMap<String, Long>();
|
||||
for (BucketStats bucket : stats.buckets()) {
|
||||
OptionalLong newest = bucket.newestGeneration();
|
||||
for (TableShardStats tableShard : stats.tableShards()) {
|
||||
OptionalLong newest = tableShard.newestSstableGeneration();
|
||||
if (newest.isPresent()) {
|
||||
targets.put(bucket.shardId(), newest.getAsLong());
|
||||
targets.put(tableShard.shardId(), newest.getAsLong());
|
||||
}
|
||||
}
|
||||
return targets;
|
||||
@@ -1,56 +0,0 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Live per-bucket LSM state, as returned by {@link LanceDbTableLsm#getLsmStats()}.
|
||||
*
|
||||
* <p>Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are the caller's to
|
||||
* compute. There is no "LSM is off" shape — that case is an empty {@link java.util.Optional},
|
||||
* because a stats object of zeros would read as measurements.
|
||||
*/
|
||||
public class LsmStats {
|
||||
private static final String CONTEXT = "lsm stats";
|
||||
|
||||
private final List<BucketStats> buckets;
|
||||
|
||||
LsmStats(List<BucketStats> buckets) {
|
||||
this.buckets = Collections.unmodifiableList(buckets);
|
||||
}
|
||||
|
||||
/** One entry per bucket. */
|
||||
public List<BucketStats> buckets() {
|
||||
return buckets;
|
||||
}
|
||||
|
||||
static LsmStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
List<BucketStats> buckets = new ArrayList<BucketStats>();
|
||||
for (JsonNode bucket : JsonFields.requiredArray(node, "buckets", CONTEXT)) {
|
||||
buckets.add(BucketStats.fromJson(bucket));
|
||||
}
|
||||
return new LsmStats(buckets);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "LsmStats{buckets=" + buckets + "}";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Live per-table-shard MemWAL state, as returned by {@link LanceDbTableMemWal#getMemWalStats()}.
|
||||
*
|
||||
* <p>Nothing here is derived: sums and differences (total SSTable bytes, WAL lag) are the caller's
|
||||
* to compute. There is no "LSM is off" shape — that case is an empty {@link java.util.Optional},
|
||||
* because a stats object of zeros would read as measurements.
|
||||
*/
|
||||
public class MemWalStats {
|
||||
private static final String CONTEXT = "memwal stats";
|
||||
|
||||
private final List<TableShardStats> tableShards;
|
||||
|
||||
MemWalStats(List<TableShardStats> tableShards) {
|
||||
this.tableShards = Collections.unmodifiableList(tableShards);
|
||||
}
|
||||
|
||||
/** One entry per table shard. */
|
||||
public List<TableShardStats> tableShards() {
|
||||
return tableShards;
|
||||
}
|
||||
|
||||
static MemWalStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
List<TableShardStats> tableShards = new ArrayList<TableShardStats>();
|
||||
for (JsonNode tableShard : JsonFields.requiredArray(node, "table_shards", CONTEXT)) {
|
||||
tableShards.add(TableShardStats.fromJson(tableShard));
|
||||
}
|
||||
return new MemWalStats(tableShards);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "MemWalStats{tableShards=" + tableShards + "}";
|
||||
}
|
||||
}
|
||||
+21
-20
@@ -27,16 +27,16 @@ import java.util.Map;
|
||||
*
|
||||
* <p>Construct via {@link #bucket}, {@link #identity}, or {@link #unsharded}, then optionally chain
|
||||
* {@link #withMaintainedIndexes} and {@link #withWriterConfigDefaults}. Install it with {@link
|
||||
* LanceDbTableLsm#setLsmWriteSpec} and remove it with {@link LanceDbTableLsm#unsetLsmWriteSpec}.
|
||||
* LanceDbTableMemWal#enableMemWal} and remove it with {@link LanceDbTableMemWal#disableMemWal}.
|
||||
*
|
||||
* <p>This is deliberately not {@code org.lance.memwal.InitializeMemWalParams}. That type is Lance's
|
||||
* own, and its maintained-index default is the opposite of this one: it defaults to maintaining
|
||||
* <em>nothing</em>, while a fresh spec here maintains <em>every</em> index. It also cannot express
|
||||
* the null that asks the server to resolve the set.
|
||||
*/
|
||||
public class LsmWriteSpec {
|
||||
public class MemWalWriteSpec {
|
||||
|
||||
private static final String CONTEXT = "get_lsm_write_spec response";
|
||||
private static final String CONTEXT = "get_memwal_spec response";
|
||||
|
||||
/** How writes are routed to MemWAL shards. */
|
||||
public enum Sharding {
|
||||
@@ -73,7 +73,7 @@ public class LsmWriteSpec {
|
||||
private final List<String> maintainedIndexes;
|
||||
private final Map<String, String> writerConfigDefaults;
|
||||
|
||||
private LsmWriteSpec(
|
||||
private MemWalWriteSpec(
|
||||
Sharding sharding,
|
||||
String column,
|
||||
Integer numBuckets,
|
||||
@@ -95,11 +95,11 @@ public class LsmWriteSpec {
|
||||
* @param column A non-nested column with a supported scalar type.
|
||||
* @param numBuckets The number of buckets, in {@code [1, 1024]}.
|
||||
*/
|
||||
public static LsmWriteSpec bucket(String column, int numBuckets) {
|
||||
public static MemWalWriteSpec bucket(String column, int numBuckets) {
|
||||
if (column == null || column.trim().isEmpty()) {
|
||||
throw new IllegalArgumentException("Column cannot be null or empty");
|
||||
}
|
||||
return new LsmWriteSpec(
|
||||
return new MemWalWriteSpec(
|
||||
Sharding.BUCKET, column, numBuckets, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
@@ -111,16 +111,17 @@ public class LsmWriteSpec {
|
||||
* with a given primary key must always produce the same {@code column} value, or upserts of that
|
||||
* key can land in different shards and a stale version can win.
|
||||
*/
|
||||
public static LsmWriteSpec identity(String column) {
|
||||
public static MemWalWriteSpec identity(String column) {
|
||||
if (column == null || column.trim().isEmpty()) {
|
||||
throw new IllegalArgumentException("Column cannot be null or empty");
|
||||
}
|
||||
return new LsmWriteSpec(Sharding.IDENTITY, column, null, null, new HashMap<String, String>());
|
||||
return new MemWalWriteSpec(
|
||||
Sharding.IDENTITY, column, null, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
/** No sharding — every write goes to a single MemWAL shard — maintaining every index. */
|
||||
public static LsmWriteSpec unsharded() {
|
||||
return new LsmWriteSpec(Sharding.UNSHARDED, null, null, null, new HashMap<String, String>());
|
||||
public static MemWalWriteSpec unsharded() {
|
||||
return new MemWalWriteSpec(Sharding.UNSHARDED, null, null, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -131,8 +132,8 @@ public class LsmWriteSpec {
|
||||
*
|
||||
* <p>Note that {@code null} and the empty list mean opposite things here.
|
||||
*/
|
||||
public LsmWriteSpec withMaintainedIndexes(List<String> maintainedIndexes) {
|
||||
return new LsmWriteSpec(
|
||||
public MemWalWriteSpec withMaintainedIndexes(List<String> maintainedIndexes) {
|
||||
return new MemWalWriteSpec(
|
||||
sharding,
|
||||
column,
|
||||
numBuckets,
|
||||
@@ -149,11 +150,11 @@ public class LsmWriteSpec {
|
||||
* max_unflushed_memtable_bytes}, and {@code enable_memtable}. Duration knobs carry an {@code _ms}
|
||||
* suffix, such as {@code max_wal_flush_interval_ms}.
|
||||
*/
|
||||
public LsmWriteSpec withWriterConfigDefaults(Map<String, String> writerConfigDefaults) {
|
||||
public MemWalWriteSpec withWriterConfigDefaults(Map<String, String> writerConfigDefaults) {
|
||||
if (writerConfigDefaults == null) {
|
||||
throw new IllegalArgumentException("writerConfigDefaults cannot be null");
|
||||
}
|
||||
return new LsmWriteSpec(
|
||||
return new MemWalWriteSpec(
|
||||
sharding,
|
||||
column,
|
||||
numBuckets,
|
||||
@@ -191,7 +192,7 @@ public class LsmWriteSpec {
|
||||
return Collections.unmodifiableMap(writerConfigDefaults);
|
||||
}
|
||||
|
||||
/** Render this spec as the {@code set_lsm_write_spec} request body. */
|
||||
/** Render this spec as the {@code enable_memwal} request body. */
|
||||
Map<String, Object> toRequestBody() {
|
||||
Map<String, Object> shardingBody = new LinkedHashMap<String, Object>();
|
||||
shardingBody.put("mode", sharding.wireName());
|
||||
@@ -211,15 +212,15 @@ public class LsmWriteSpec {
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild a spec from a {@code get_lsm_write_spec} response body.
|
||||
* Rebuild a spec from a {@code get_memwal_spec} response body.
|
||||
*
|
||||
* <p>A null {@code maintained_indexes} selects every index the table has; an empty list selects
|
||||
* none.
|
||||
*/
|
||||
static LsmWriteSpec fromJson(JsonNode node) {
|
||||
static MemWalWriteSpec fromJson(JsonNode node) {
|
||||
JsonNode shardingNode = node.get("sharding");
|
||||
if (shardingNode == null || shardingNode.get("mode") == null) {
|
||||
throw new IllegalStateException("get_lsm_write_spec response has no sharding mode");
|
||||
throw new IllegalStateException("get_memwal_spec response has no sharding mode");
|
||||
}
|
||||
Sharding sharding = Sharding.fromWireName(shardingNode.get("mode").asText());
|
||||
|
||||
@@ -252,12 +253,12 @@ public class LsmWriteSpec {
|
||||
.forEachRemaining(name -> defaults.put(name, defaultsNode.get(name).asText()));
|
||||
}
|
||||
|
||||
return new LsmWriteSpec(sharding, column, numBuckets, maintainedIndexes, defaults);
|
||||
return new MemWalWriteSpec(sharding, column, numBuckets, maintainedIndexes, defaults);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "LsmWriteSpec{sharding="
|
||||
return "MemWalWriteSpec{sharding="
|
||||
+ sharding
|
||||
+ ", column="
|
||||
+ column
|
||||
@@ -37,7 +37,7 @@ public class MemtableStats {
|
||||
this.indexes = Collections.unmodifiableList(indexes);
|
||||
}
|
||||
|
||||
/** The generation this memtable will become once sealed. */
|
||||
/** The generation this memtable will become once frozen. */
|
||||
public long generation() {
|
||||
return generation;
|
||||
}
|
||||
|
||||
+8
-8
@@ -17,21 +17,21 @@ import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/** One flushed L0 generation. */
|
||||
public class GenerationStats {
|
||||
/** One SSTable. */
|
||||
public class SsTableStats {
|
||||
private static final String CONTEXT = "generation stats";
|
||||
|
||||
private final long generation;
|
||||
private final long bytes;
|
||||
private final Long rows;
|
||||
|
||||
GenerationStats(long generation, long bytes, Long rows) {
|
||||
SsTableStats(long generation, long bytes, Long rows) {
|
||||
this.generation = generation;
|
||||
this.bytes = bytes;
|
||||
this.rows = rows;
|
||||
}
|
||||
|
||||
/** The generation number. Increases as memtables are sealed into L0. */
|
||||
/** The generation number. Increases as memtables are frozen into SSTables. */
|
||||
public long generation() {
|
||||
return generation;
|
||||
}
|
||||
@@ -42,16 +42,16 @@ public class GenerationStats {
|
||||
}
|
||||
|
||||
/**
|
||||
* Rows in this generation, present only when {@code includeGenerationRows} was requested. Off by
|
||||
* Rows in this generation, present only when {@code includeSstableRows} was requested. Off by
|
||||
* default because each count opens an uncached Lance dataset.
|
||||
*/
|
||||
public OptionalLong rows() {
|
||||
return rows == null ? OptionalLong.empty() : OptionalLong.of(rows);
|
||||
}
|
||||
|
||||
static GenerationStats fromJson(JsonNode node) {
|
||||
static SsTableStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
return new GenerationStats(
|
||||
return new SsTableStats(
|
||||
JsonFields.requiredLong(node, "generation", CONTEXT),
|
||||
JsonFields.requiredLong(node, "bytes", CONTEXT),
|
||||
JsonFields.optionalLong(node, "rows", CONTEXT));
|
||||
@@ -59,6 +59,6 @@ public class GenerationStats {
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "GenerationStats{generation=" + generation + ", bytes=" + bytes + ", rows=" + rows + "}";
|
||||
return "SsTableStats{generation=" + generation + ", bytes=" + bytes + ", rows=" + rows + "}";
|
||||
}
|
||||
}
|
||||
+31
-29
@@ -22,11 +22,11 @@ import java.util.Optional;
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/**
|
||||
* Live state of one bucket. A table is N buckets on one node; flattening to a single number hides
|
||||
* the one hot bucket that is usually why someone opened this endpoint.
|
||||
* Live state of one table shard. A table is N table shards on one node; flattening to a single
|
||||
* number hides the one hot table shard that is usually why someone opened this endpoint.
|
||||
*/
|
||||
public class BucketStats {
|
||||
private static final String CONTEXT = "bucket stats";
|
||||
public class TableShardStats {
|
||||
private static final String CONTEXT = "tableShard stats";
|
||||
|
||||
private final String shardId;
|
||||
private final String status;
|
||||
@@ -35,11 +35,11 @@ public class BucketStats {
|
||||
private final long currentGeneration;
|
||||
private final long replayAfterWalEntryPosition;
|
||||
private final long walEntryPositionLastSeen;
|
||||
private final List<GenerationStats> generations;
|
||||
private final List<SsTableStats> sstables;
|
||||
private final boolean compacting;
|
||||
private final List<MemtableStats> memtables;
|
||||
|
||||
BucketStats(
|
||||
TableShardStats(
|
||||
String shardId,
|
||||
String status,
|
||||
long writerEpoch,
|
||||
@@ -47,7 +47,7 @@ public class BucketStats {
|
||||
long currentGeneration,
|
||||
long replayAfterWalEntryPosition,
|
||||
long walEntryPositionLastSeen,
|
||||
List<GenerationStats> generations,
|
||||
List<SsTableStats> sstables,
|
||||
boolean compacting,
|
||||
List<MemtableStats> memtables) {
|
||||
this.shardId = shardId;
|
||||
@@ -57,12 +57,12 @@ public class BucketStats {
|
||||
this.currentGeneration = currentGeneration;
|
||||
this.replayAfterWalEntryPosition = replayAfterWalEntryPosition;
|
||||
this.walEntryPositionLastSeen = walEntryPositionLastSeen;
|
||||
this.generations = Collections.unmodifiableList(generations);
|
||||
this.sstables = Collections.unmodifiableList(sstables);
|
||||
this.compacting = compacting;
|
||||
this.memtables = memtables == null ? null : Collections.unmodifiableList(memtables);
|
||||
}
|
||||
|
||||
/** The shard this bucket writes. */
|
||||
/** The id of this table shard. */
|
||||
public String shardId() {
|
||||
return shardId;
|
||||
}
|
||||
@@ -100,13 +100,13 @@ public class BucketStats {
|
||||
return walEntryPositionLastSeen;
|
||||
}
|
||||
|
||||
/** Flushed L0 generations not yet merged into the base table. */
|
||||
public List<GenerationStats> generations() {
|
||||
return generations;
|
||||
/** SSTables not yet merged into the base table. */
|
||||
public List<SsTableStats> sstables() {
|
||||
return sstables;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a pass owns this bucket's compaction latch right now. Says <em>a</em> driver is
|
||||
* Whether a pass owns this table shard's compaction latch right now. Says <em>a</em> driver is
|
||||
* running, not <em>whose</em>, and the latch is held from dispatch — including while the pass
|
||||
* queues for a pod-wide compactor permit. Read it as "do not pile on", never as "mine is
|
||||
* progressing".
|
||||
@@ -115,15 +115,17 @@ public class BucketStats {
|
||||
return compacting;
|
||||
}
|
||||
|
||||
/** Oldest first, active last. Empty for a {@code "Sealed"} bucket, whose state is torn down. */
|
||||
/**
|
||||
* Oldest first, active last. Empty for a {@code "Sealed"} table shard, whose state is torn down.
|
||||
*/
|
||||
public Optional<List<MemtableStats>> memtables() {
|
||||
return Optional.ofNullable(memtables);
|
||||
}
|
||||
|
||||
/** The newest flushed generation, or empty when L0 is empty. */
|
||||
OptionalLong newestGeneration() {
|
||||
/** The newest SSTable generation, or empty when the tier is empty. */
|
||||
OptionalLong newestSstableGeneration() {
|
||||
OptionalLong newest = OptionalLong.empty();
|
||||
for (GenerationStats generation : generations) {
|
||||
for (SsTableStats generation : sstables) {
|
||||
if (!newest.isPresent() || generation.generation() > newest.getAsLong()) {
|
||||
newest = OptionalLong.of(generation.generation());
|
||||
}
|
||||
@@ -132,15 +134,15 @@ public class BucketStats {
|
||||
}
|
||||
|
||||
/**
|
||||
* How many generations at or below {@code target} are still in L0.
|
||||
* How many SSTables at or below {@code target} are still uncompacted.
|
||||
*
|
||||
* <p>A count, not a boolean: one pass drains a bounded prefix rather than the whole target set,
|
||||
* so a boolean would read as "no progress" for every pass but the last. Compaction drains
|
||||
* oldest-first, so this decreases monotonically.
|
||||
*/
|
||||
long outstandingGenerations(long target) {
|
||||
long outstandingSstables(long target) {
|
||||
long count = 0;
|
||||
for (GenerationStats generation : generations) {
|
||||
for (SsTableStats generation : sstables) {
|
||||
if (generation.generation() <= target) {
|
||||
count++;
|
||||
}
|
||||
@@ -148,11 +150,11 @@ public class BucketStats {
|
||||
return count;
|
||||
}
|
||||
|
||||
static BucketStats fromJson(JsonNode node) {
|
||||
static TableShardStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
List<GenerationStats> generations = new ArrayList<GenerationStats>();
|
||||
for (JsonNode generation : JsonFields.requiredArray(node, "generations", CONTEXT)) {
|
||||
generations.add(GenerationStats.fromJson(generation));
|
||||
List<SsTableStats> sstables = new ArrayList<SsTableStats>();
|
||||
for (JsonNode generation : JsonFields.requiredArray(node, "sstables", CONTEXT)) {
|
||||
sstables.add(SsTableStats.fromJson(generation));
|
||||
}
|
||||
|
||||
JsonNode memtablesNode = JsonFields.optionalArray(node, "memtables", CONTEXT);
|
||||
@@ -164,7 +166,7 @@ public class BucketStats {
|
||||
}
|
||||
}
|
||||
|
||||
return new BucketStats(
|
||||
return new TableShardStats(
|
||||
JsonFields.requiredText(node, "shard_id", CONTEXT),
|
||||
JsonFields.requiredText(node, "status", CONTEXT),
|
||||
JsonFields.requiredLong(node, "writer_epoch", CONTEXT),
|
||||
@@ -172,21 +174,21 @@ public class BucketStats {
|
||||
JsonFields.requiredLong(node, "current_generation", CONTEXT),
|
||||
JsonFields.requiredLong(node, "replay_after_wal_entry_position", CONTEXT),
|
||||
JsonFields.requiredLong(node, "wal_entry_position_last_seen", CONTEXT),
|
||||
generations,
|
||||
sstables,
|
||||
JsonFields.requiredBoolean(node, "compacting", CONTEXT),
|
||||
memtables);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "BucketStats{shardId="
|
||||
return "TableShardStats{shardId="
|
||||
+ shardId
|
||||
+ ", status="
|
||||
+ status
|
||||
+ ", currentGeneration="
|
||||
+ currentGeneration
|
||||
+ ", generations="
|
||||
+ generations
|
||||
+ ", sstables="
|
||||
+ sstables
|
||||
+ ", compacting="
|
||||
+ compacting
|
||||
+ "}";
|
||||
+151
-148
@@ -46,12 +46,12 @@ import static org.junit.jupiter.api.Assertions.*;
|
||||
* <p>The wire assertions mirror the Rust mocked-endpoint tests in {@code
|
||||
* rust/lancedb/src/remote/table.rs}, which are the contract these routes have to match.
|
||||
*/
|
||||
public class LanceDbTableLsmTest {
|
||||
public class LanceDbTableMemWalTest {
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
|
||||
private HttpServer server;
|
||||
private LanceDbRestClient client;
|
||||
private LanceDbTableLsm lsm;
|
||||
private LanceDbTableMemWal memWal;
|
||||
|
||||
private final List<String> requestPaths = Collections.synchronizedList(new ArrayList<String>());
|
||||
private final List<String> requestBodies = Collections.synchronizedList(new ArrayList<String>());
|
||||
@@ -101,7 +101,7 @@ public class LanceDbTableLsmTest {
|
||||
.database("test-db")
|
||||
.endpoint("http://127.0.0.1:" + server.getAddress().getPort())
|
||||
.buildRestClient();
|
||||
lsm = new LanceDbTableLsm(client, "my_table");
|
||||
memWal = new LanceDbTableMemWal(client, "my_table");
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
@@ -115,12 +115,12 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecUnsharded() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalUnsharded() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded());
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded());
|
||||
|
||||
assertEquals("/v1/table/my_table/set_lsm_write_spec/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/enable_memwal/", requestPaths.get(0));
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("unsharded", body.get("sharding").get("mode").asText());
|
||||
assertFalse(body.get("sharding").has("column"));
|
||||
@@ -128,11 +128,11 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecBucket() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalBucket() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(
|
||||
LsmWriteSpec.bucket("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
|
||||
memWal.enableMemWal(
|
||||
MemWalWriteSpec.bucket("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("bucket", body.get("sharding").get("mode").asText());
|
||||
@@ -143,10 +143,10 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecIdentity() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalIdentity() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.identity("tenant"));
|
||||
memWal.enableMemWal(MemWalWriteSpec.identity("tenant"));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("identity", body.get("sharding").get("mode").asText());
|
||||
@@ -160,55 +160,55 @@ public class LanceDbTableLsmTest {
|
||||
*/
|
||||
@Test
|
||||
public void testMaintainedIndexesNullAndEmptyAreDistinctOnTheWire() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded());
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded());
|
||||
JsonNode fresh = MAPPER.readTree(requestBodies.get(0));
|
||||
assertTrue(fresh.has("maintained_indexes"), "the key must be present");
|
||||
assertTrue(fresh.get("maintained_indexes").isNull(), "a fresh spec sends null, not []");
|
||||
|
||||
lsm.setLsmWriteSpec(
|
||||
LsmWriteSpec.unsharded().withMaintainedIndexes(Collections.<String>emptyList()));
|
||||
memWal.enableMemWal(
|
||||
MemWalWriteSpec.unsharded().withMaintainedIndexes(Collections.<String>emptyList()));
|
||||
JsonNode none = MAPPER.readTree(requestBodies.get(1));
|
||||
assertTrue(none.get("maintained_indexes").isArray());
|
||||
assertEquals(0, none.get("maintained_indexes").size());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecWriterConfigDefaults() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
public void testEnableMemWalWriterConfigDefaults() throws Exception {
|
||||
enqueue("enable_memwal", 200, "");
|
||||
|
||||
Map<String, String> defaults = new HashMap<String, String>();
|
||||
defaults.put("max_memtable_rows", "50000");
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded().withWriterConfigDefaults(defaults));
|
||||
memWal.enableMemWal(MemWalWriteSpec.unsharded().withWriterConfigDefaults(defaults));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("50000", body.get("writer_config_defaults").get("max_memtable_rows").asText());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testUnsetLsmWriteSpec() {
|
||||
enqueue("unset_lsm_write_spec", 200, "");
|
||||
public void testDisableMemWal() {
|
||||
enqueue("disable_memwal", 200, "");
|
||||
|
||||
lsm.unsetLsmWriteSpec();
|
||||
memWal.disableMemWal();
|
||||
|
||||
assertEquals("/v1/table/my_table/unset_lsm_write_spec/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/disable_memwal/", requestPaths.get(0));
|
||||
assertEquals("", requestBodies.get(0));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpec() {
|
||||
public void testGetMemWalSpec() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"bucket\",\"column\":\"id\","
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"bucket\",\"column\":\"id\","
|
||||
+ "\"num_buckets\":16},\"maintained_indexes\":[\"id_idx\"],"
|
||||
+ "\"writer_config_defaults\":{\"durable_write\":\"true\"}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertEquals(LsmWriteSpec.Sharding.BUCKET, spec.get().sharding());
|
||||
assertEquals(MemWalWriteSpec.Sharding.BUCKET, spec.get().sharding());
|
||||
assertEquals("id", spec.get().column());
|
||||
assertEquals(Integer.valueOf(16), spec.get().numBuckets());
|
||||
assertEquals(Arrays.asList("id_idx"), spec.get().maintainedIndexes());
|
||||
@@ -217,28 +217,28 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
/** A null selection, every index the table has, reads back as null rather than an empty list. */
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRoundTripsEveryIndexSelection() {
|
||||
public void testGetMemWalSpecRoundTripsEveryIndexSelection() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":null,\"writer_config_defaults\":{}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertNull(spec.get().maintainedIndexes());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRoundTripsNoIndexSelection() {
|
||||
public void testGetMemWalSpecRoundTripsNoIndexSelection() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":[],\"writer_config_defaults\":{}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
Optional<MemWalWriteSpec> spec = memWal.getMemWalSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertEquals(Collections.emptyList(), spec.get().maintainedIndexes());
|
||||
@@ -249,32 +249,32 @@ public class LanceDbTableLsmTest {
|
||||
* rejected: reading a string or an object as null would turn a malformed response into a policy.
|
||||
*/
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRejectsANonArrayMaintainedIndexes() {
|
||||
public void testGetMemWalSpecRejectsANonArrayMaintainedIndexes() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":\"id_idx\",\"writer_config_defaults\":{}}}");
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> lsm.getLsmWriteSpec());
|
||||
assertThrows(IllegalStateException.class, () -> memWal.getMemWalSpec());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecRejectsNonStringIndexNames() {
|
||||
public void testGetMemWalSpecRejectsNonStringIndexNames() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
"get_memwal_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
"{\"memwal_spec\":{\"sharding\":{\"mode\":\"unsharded\"},"
|
||||
+ "\"maintained_indexes\":[7],\"writer_config_defaults\":{}}}");
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> lsm.getLsmWriteSpec());
|
||||
assertThrows(IllegalStateException.class, () -> memWal.getMemWalSpec());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecAbsent() {
|
||||
enqueue("get_lsm_write_spec", 200, "{\"lsm_write_spec\":null}");
|
||||
public void testGetMemWalSpecAbsent() {
|
||||
enqueue("get_memwal_spec", 200, "{\"memwal_spec\":null}");
|
||||
|
||||
assertFalse(lsm.getLsmWriteSpec().isPresent());
|
||||
assertFalse(memWal.getMemWalSpec().isPresent());
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -282,15 +282,15 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testGetLsmStats() throws Exception {
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
public void testGetMemWalStats() throws Exception {
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
|
||||
Optional<LsmStats> got = lsm.getLsmStats(true);
|
||||
Optional<MemWalStats> got = memWal.getMemWalStats(true);
|
||||
|
||||
assertEquals("/v1/table/my_table/get_lsm_stats/", requestPaths.get(0));
|
||||
assertTrue(MAPPER.readTree(requestBodies.get(0)).get("include_generation_rows").asBoolean());
|
||||
assertEquals("/v1/table/my_table/get_memwal_stats/", requestPaths.get(0));
|
||||
assertTrue(MAPPER.readTree(requestBodies.get(0)).get("include_sstable_rows").asBoolean());
|
||||
assertTrue(got.isPresent());
|
||||
BucketStats decoded = got.get().buckets().get(0);
|
||||
TableShardStats decoded = got.get().tableShards().get(0);
|
||||
assertEquals("shard-0", decoded.shardId());
|
||||
assertEquals("Active", decoded.status());
|
||||
assertEquals(1, decoded.writerEpoch());
|
||||
@@ -298,30 +298,30 @@ public class LanceDbTableLsmTest {
|
||||
assertEquals(9, decoded.currentGeneration());
|
||||
assertFalse(decoded.compacting());
|
||||
assertEquals(Arrays.asList(7L, 8L), generationNumbers(decoded));
|
||||
assertEquals(1024, decoded.generations().get(0).bytes());
|
||||
assertFalse(decoded.generations().get(0).rows().isPresent(), "rows absent unless requested");
|
||||
assertEquals(1024, decoded.sstables().get(0).bytes());
|
||||
assertFalse(decoded.sstables().get(0).rows().isPresent(), "rows absent unless requested");
|
||||
assertFalse(decoded.memtables().isPresent(), "absent memtables stay absent");
|
||||
}
|
||||
|
||||
/** The optional fields decode when the server does send them. */
|
||||
@Test
|
||||
public void testGetLsmStatsDecodesOptionalFields() {
|
||||
public void testGetMemWalStatsDecodesOptionalFields() {
|
||||
enqueue(
|
||||
"get_lsm_stats",
|
||||
"get_memwal_stats",
|
||||
200,
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":3,\"wal_entry_position_last_seen\":11,"
|
||||
+ "\"generations\":[{\"generation\":7,\"bytes\":1024,\"rows\":42}],"
|
||||
+ "\"sstables\":[{\"generation\":7,\"bytes\":1024,\"rows\":42}],"
|
||||
+ "\"compacting\":true,\"memtables\":[{\"generation\":8,\"rows\":5,"
|
||||
+ "\"bytes\":64,\"batches\":2,\"indexes\":[\"id_idx\"]}]}]}}");
|
||||
|
||||
BucketStats decoded = lsm.getLsmStats(true).get().buckets().get(0);
|
||||
TableShardStats decoded = memWal.getMemWalStats(true).get().tableShards().get(0);
|
||||
|
||||
assertEquals(3, decoded.replayAfterWalEntryPosition());
|
||||
assertEquals(11, decoded.walEntryPositionLastSeen());
|
||||
assertTrue(decoded.compacting());
|
||||
assertEquals(42, decoded.generations().get(0).rows().getAsLong());
|
||||
assertEquals(42, decoded.sstables().get(0).rows().getAsLong());
|
||||
assertTrue(decoded.memtables().isPresent());
|
||||
MemtableStats memtable = decoded.memtables().get().get(0);
|
||||
assertEquals(8, memtable.generation());
|
||||
@@ -332,19 +332,19 @@ public class LanceDbTableLsmTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmStatsAbsentWhenLsmDisabled() {
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
public void testGetMemWalStatsAbsentWhenMemWalDisabled() {
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
assertFalse(lsm.getLsmStats().isPresent());
|
||||
assertFalse(memWal.getMemWalStats().isPresent());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmStatsDefaultsToExcludingGenerationRows() throws Exception {
|
||||
enqueue("get_lsm_stats", 200, stats());
|
||||
public void testGetMemWalStatsDefaultsToExcludingGenerationRows() throws Exception {
|
||||
enqueue("get_memwal_stats", 200, stats());
|
||||
|
||||
lsm.getLsmStats();
|
||||
memWal.getMemWalStats();
|
||||
|
||||
assertFalse(MAPPER.readTree(requestBodies.get(0)).get("include_generation_rows").asBoolean());
|
||||
assertFalse(MAPPER.readTree(requestBodies.get(0)).get("include_sstable_rows").asBoolean());
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -353,22 +353,22 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
@Test
|
||||
public void testFlushAndCompactRoutes() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("compact_lsm", 200, "");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("compact_memwal", 200, "");
|
||||
|
||||
lsm.flushLsm();
|
||||
lsm.compactLsm();
|
||||
memWal.flushMemWal();
|
||||
memWal.compactMemWal();
|
||||
|
||||
assertEquals("/v1/table/my_table/flush_lsm/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/compact_lsm/", requestPaths.get(1));
|
||||
assertEquals("/v1/table/my_table/flush_memwal/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/compact_memwal/", requestPaths.get(1));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testHttpErrorCarriesStatus() {
|
||||
enqueue("flush_lsm", 404, "no such table");
|
||||
enqueue("flush_memwal", 404, "no such table");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.flushLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.flushMemWal());
|
||||
assertEquals(404, e.statusCode());
|
||||
}
|
||||
|
||||
@@ -377,97 +377,98 @@ public class LanceDbTableLsmTest {
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testCheckpointReturnsWhenLsmDisabled() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
public void testCheckpointReturnsWhenMemWalDisabled() {
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"), "nothing to compact when the LSM path is off");
|
||||
assertEquals(0, countCalls("compact_memwal"), "nothing to compact when the LSM path is off");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointReturnsWhenNoGenerationsOutstanding() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
// A bucket with no L0 generations yields no target, so the drain never starts.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
// A table shard with no SSTables yields no target, so the drain never starts.
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"));
|
||||
assertEquals(0, countCalls("compact_memwal"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointConvergesOnceTargetGenerationsAreGone() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
// Watermark read: shard-0 holds generations 7 and 8, so target = 8.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
// Watermark read: shard-0 holds sstables 7 and 8, so target = 8.
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
// First drain poll: both still outstanding, nothing compacting -> dispatch a pass.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 7L, 8L)));
|
||||
// Second drain poll: drained past the target -> done.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 9L)));
|
||||
enqueue("compact_lsm", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 9L)));
|
||||
enqueue("compact_memwal", 200, "");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(1, countCalls("compact_lsm"), "one pass dispatched");
|
||||
assertEquals(3, countCalls("get_lsm_stats"), "watermark read plus two drain polls");
|
||||
assertEquals(1, countCalls("compact_memwal"), "one pass dispatched");
|
||||
assertEquals(3, countCalls("get_memwal_stats"), "watermark read plus two drain polls");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointDoesNotPileOnWhileEveryTargetBucketIsCompacting() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", true, 4L)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", true, 4L)));
|
||||
// Still compacting on the first poll, so no pass is dispatched; then it drains.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", true, 4L)));
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 5L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", true, 4L)));
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false, 5L)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"), "a latched bucket is left alone");
|
||||
assertEquals(0, countCalls("compact_memwal"), "a latched table shard is left alone");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointRetriesFromFlushAfterLostClaim() {
|
||||
// 421 on the watermark read: the node lost its claim, so the whole thing restarts
|
||||
// from flush rather than retrying the read in place.
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 421, "no claim");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 421, "no claim");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(2, countCalls("flush_lsm"), "re-issued from flush");
|
||||
assertEquals(2, countCalls("flush_memwal"), "re-issued from flush");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointRetriesRetryableStatusInPlace() {
|
||||
enqueue("flush_lsm", 429, "latch held");
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
enqueue("flush_memwal", 429, "latch held");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, stats(tableShard("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(2, countCalls("flush_lsm"), "429 retried in place, not re-issued");
|
||||
assertEquals(2, countCalls("flush_memwal"), "429 retried in place, not re-issued");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointPropagatesTerminalStatus() {
|
||||
enqueue("flush_lsm", 400, "bad request");
|
||||
enqueue("flush_memwal", 400, "bad request");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.checkpointLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.checkpointMemWal());
|
||||
assertEquals(400, e.statusCode());
|
||||
assertEquals(1, countCalls("flush_lsm"), "a terminal status is not retried");
|
||||
assertEquals(1, countCalls("flush_memwal"), "a terminal status is not retried");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointGivesUpAfterRepeatedLostClaims() {
|
||||
enqueue("flush_lsm", 421, "no claim");
|
||||
enqueue("flush_memwal", 421, "no claim");
|
||||
|
||||
IllegalStateException e = assertThrows(IllegalStateException.class, () -> lsm.checkpointLsm());
|
||||
IllegalStateException e =
|
||||
assertThrows(IllegalStateException.class, () -> memWal.checkpointMemWal());
|
||||
assertTrue(e.getMessage().contains("kept losing its claim"), e.getMessage());
|
||||
assertEquals(4, countCalls("flush_lsm"), "the initial attempt plus MAX_REISSUES");
|
||||
assertEquals(4, countCalls("flush_memwal"), "the initial attempt plus MAX_REISSUES");
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -476,50 +477,51 @@ public class LanceDbTableLsmTest {
|
||||
|
||||
/**
|
||||
* A stats payload that does not decode must fail closed. Every one of these bodies used to be
|
||||
* read as "no buckets", which is indistinguishable from a drained table, so {@code checkpointLsm}
|
||||
* reported convergence for a checkpoint that never ran.
|
||||
* read as "no table shards", which is indistinguishable from a drained table, so {@code
|
||||
* checkpointMemWal} reported convergence for a checkpoint that never ran.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckpointRejectsMalformedStats() {
|
||||
Map<String, String> malformed = new LinkedHashMap<String, String>();
|
||||
malformed.put("no response body at all", "");
|
||||
malformed.put("stats object with no buckets", "{\"lsm_stats\":{}}");
|
||||
malformed.put("bucket missing its required fields", "{\"lsm_stats\":{\"buckets\":[{}]}}");
|
||||
malformed.put("stats object with no table shards", "{\"memwal_stats\":{}}");
|
||||
malformed.put(
|
||||
"bucket missing generations",
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"table shard missing its required fields", "{\"memwal_stats\":{\"table_shards\":[{}]}}");
|
||||
malformed.put(
|
||||
"table shard missing sstables",
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
|
||||
+ "\"compacting\":false}]}}");
|
||||
malformed.put(
|
||||
"generation with a non-numeric generation number",
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
"{\"memwal_stats\":{\"table_shards\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
|
||||
+ "\"generations\":[{\"generation\":\"7\",\"bytes\":1024}],"
|
||||
+ "\"sstables\":[{\"generation\":\"7\",\"bytes\":1024}],"
|
||||
+ "\"compacting\":false}]}}");
|
||||
|
||||
for (Map.Entry<String, String> each : malformed.entrySet()) {
|
||||
setUpFresh();
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, each.getValue());
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, each.getValue());
|
||||
|
||||
assertThrows(
|
||||
IllegalStateException.class,
|
||||
() -> lsm.checkpointLsm(),
|
||||
() -> memWal.checkpointMemWal(),
|
||||
each.getKey() + " must not report convergence");
|
||||
}
|
||||
}
|
||||
|
||||
/** The one shape that legitimately means "this table has no LSM write path". */
|
||||
/** The one shape that legitimately means "this table has no MemWAL write path". */
|
||||
@Test
|
||||
public void testCheckpointTreatsNullStatsAsNotWalBacked() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
enqueue("flush_memwal", 200, "");
|
||||
enqueue("get_memwal_stats", 200, "{\"memwal_stats\":null}");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
memWal.checkpointMemWal();
|
||||
|
||||
assertEquals(1, countCalls("get_lsm_stats"));
|
||||
assertEquals(1, countCalls("get_memwal_stats"));
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
@@ -529,40 +531,41 @@ public class LanceDbTableLsmTest {
|
||||
/**
|
||||
* The transport must not retry on the checkpoint loop's behalf. Apache HttpClient's default
|
||||
* strategy retries exactly 429 and 503 — the two statuses {@code isRetryable} owns — which
|
||||
* doubled every budget here and also retried {@code compact_lsm} in place, where the loop is
|
||||
* doubled every budget here and also retried {@code compact_memwal} in place, where the loop is
|
||||
* built to fall through to a fresh stats poll instead.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckpointRetryBudgetIsNotDoubledByTheTransport() {
|
||||
enqueue("flush_lsm", 429, "latch held");
|
||||
enqueue("flush_memwal", 429, "latch held");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.checkpointLsm());
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> memWal.checkpointMemWal());
|
||||
|
||||
assertEquals(429, e.statusCode(), "the exhausted budget propagates the last error as itself");
|
||||
assertEquals(9, countCalls("flush_lsm"), "the initial request plus MAX_RETRIES, and no more");
|
||||
assertEquals(
|
||||
9, countCalls("flush_memwal"), "the initial request plus MAX_RETRIES, and no more");
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// harness
|
||||
// ===========================================================================
|
||||
|
||||
private static List<Long> generationNumbers(BucketStats bucket) {
|
||||
private static List<Long> generationNumbers(TableShardStats tableShard) {
|
||||
List<Long> numbers = new ArrayList<Long>();
|
||||
for (GenerationStats generation : bucket.generations()) {
|
||||
for (SsTableStats generation : tableShard.sstables()) {
|
||||
numbers.add(generation.generation());
|
||||
}
|
||||
return numbers;
|
||||
}
|
||||
|
||||
/** Build an {@code lsm_stats} response body from bucket fragments. */
|
||||
private static String stats(String... buckets) {
|
||||
return "{\"lsm_stats\":{\"buckets\":[" + String.join(",", buckets) + "]}}";
|
||||
/** Build a {@code memwal_stats} response body from table shard fragments. */
|
||||
private static String stats(String... tableShards) {
|
||||
return "{\"memwal_stats\":{\"table_shards\":[" + String.join(",", tableShards) + "]}}";
|
||||
}
|
||||
|
||||
private static String bucket(String shardId, boolean compacting, Long... generations) {
|
||||
private static String tableShard(String shardId, boolean compacting, Long... sstables) {
|
||||
StringBuilder gens = new StringBuilder();
|
||||
for (Long generation : generations) {
|
||||
for (Long generation : sstables) {
|
||||
if (gens.length() > 0) {
|
||||
gens.append(",");
|
||||
}
|
||||
@@ -572,7 +575,7 @@ public class LanceDbTableLsmTest {
|
||||
+ shardId
|
||||
+ "\",\"status\":\"Active\",\"writer_epoch\":1,\"manifest_version\":2,"
|
||||
+ "\"current_generation\":9,\"replay_after_wal_entry_position\":0,"
|
||||
+ "\"wal_entry_position_last_seen\":0,\"generations\":["
|
||||
+ "\"wal_entry_position_last_seen\":0,\"sstables\":["
|
||||
+ gens
|
||||
+ "],\"compacting\":"
|
||||
+ compacting
|
||||
@@ -597,7 +600,7 @@ public class LanceDbTableLsmTest {
|
||||
return requestPaths.stream().filter(path -> operationOf(path).equals(operation)).count();
|
||||
}
|
||||
|
||||
/** {@code /v1/table/my_table/flush_lsm/} -> {@code flush_lsm}. */
|
||||
/** {@code /v1/table/my_table/flush_memwal/} -> {@code flush_memwal}. */
|
||||
private static String operationOf(String path) {
|
||||
String[] segments = path.split("/");
|
||||
return segments.length == 0 ? "" : segments[segments.length - 1];
|
||||
Reference in New Issue
Block a user