mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-20 13:08:23 +00:00
feat: bring the MemWAL LSM surface to parity across the SDKs (#3962)
## Why Four of the eight LSM methods are **remote-only in the core**. `impl BaseTable for NativeTable` implements only `set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`, `compact_lsm` and `get_lsm_stats` fall through to trait defaults returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and `checkpoint_lsm` is built on all three. That explains the state of the bindings: Node had bound the four that work against a local table and stopped, so a Cloud user could install an LSM write spec but had no way to observe fresh-tier state or drive a checkpoint. Java had none of it at all. | SDK | set/unset/get spec | closeWriters | flush | compact | getStats | checkpoint | |---|---|---|---|---|---|---| | Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Node *(before)* | ✅ | ✅ | — | — | — | — | | **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** | | Java *(before)* | — | — | — | — | — | — | | **Java (after)** | **new** | n/a | **new** | **new** | **new** | **new** | Go and C are separate repos and are out of scope here. `closeLsmWriters` drains cached in-process shard writers, so it has no meaning for Java, which is a pure REST client. ## Node Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and `getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats` / `MemtableStats` objects — typed rather than a JSON blob, matching the existing `LsmWriteSpec` object in the same file, with `u64` cast to `i64` per that file's convention. Because these four are remote-only, the new tests assert each binding reaches the core and surfaces `NotSupported` against a local table. That covers the wiring; behavior against a real endpoint stays covered by the mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`. ## Python No new methods. All eight are on `LanceTable`, `AsyncTable` and `RemoteTable` — the last four landed on the sync `RemoteTable` in #3961, which is merged into this branch. What was missing here was reachability. `LsmWriteSpec` was importable only from the private `lancedb._lancedb`, appearing in `table.py` solely under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no mention of it, which per the repo's docs guidance means it rendered nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in `__all__`, and documented. ## Java Java reaches LanceDB purely over REST through the generated Lance Namespace client, and these routes are not in that spec, so they are issued through a small dedicated client rather than added to the spec. That call is revisitable — LSM is one of four unspecified route families alongside `multipart_write`, `page_cache/prewarm` and `branches/diff|merge`. If those are ever regularized into the spec as a group, `LanceDbTableLsm` is one file that gets deleted. `LsmWriteSpec` here is deliberately **not** `org.lance.memwal.InitializeMemWalParams`. That type defaults to maintaining *no* indexes where a spec here defaults to maintaining *every* index, and it cannot express the `null` that asks the server to resolve the set: | Value | On the wire | Meaning | |---|---|---| | unset (null) | `null` | Server resolves **every** maintainable index | | `Collections.emptyList()` | `[]` | Maintain **none** | | `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those | A dedicated test pins null and `[]` as distinct on the wire, since collapsing them is the failure mode that motivated a LanceDB-owned type. `checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs` with its constants and status semantics intact: 429/503 retried in place against an 8-budget, 421 restarting from flush against a 3-budget, 5s poll, and a target watermark fixed after the seal so it terminates under write load. `getLsmStats` returns typed `LsmStats` / `BucketStats` / `GenerationStats` / `MemtableStats`, mirroring the Rust structs in `rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes. Decoding is strict — see below. ## Review feedback Both gatekeeper findings were real. Each was reproduced against the scripted test server first, and each fix ships with the reproducer as a regression test. **The transport was doubling every checkpoint retry budget.** `HttpClients.createDefault()` installs Apache's default response retry strategy, whose retryable-status list is exactly 429 and 503 — the two statuses `isRetryable` owns. A 429 held against `flush_lsm` issued **18** wire requests where the loop intends 9, and `compact_lsm` was retried in place despite the loop being built to fall through to a fresh stats poll instead. Timing confirmed the mechanism: that run took 25.4s ≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry interval. Automatic retries are now disabled, so the checkpoint loop is the sole owner of the 421/429/503 transitions. A side effect worth noting: `testCheckpointRetriesRetryableStatusInPlace` was passing on a transport-absorbed 429 and never reaching `issue()`'s retry branch at all. It now exercises the real path. **Stats decoding failed open.** `getLsmStats` read the response with Jackson's `path()`, which yields a missing node that iterates as an empty array — making "malformed" indistinguishable from "no buckets", which is indistinguishable from "drained". Four separate payloads made `checkpointLsm()` report convergence for a checkpoint that never ran: | Response | Before | Now | |---|---|---| | `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ | | `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` | | empty response body | **reported success** | `IllegalStateException` | | bucket missing required fields | **reported success** | `IllegalStateException` | The empty-body row is the one to weight: a proxy 200 with no body is a realistic production event, and it silently reported a checkpoint that never happened. Decoding is now strict and fails closed, matching the serde contract on the Rust side exactly. One deliberate deviation from the review comment, which asked that *only* explicit JSON `null` count as disabled: Rust has `#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to `None` there too. Java now matches that. It is an absent-or-malformed **`buckets`** that fails closed, which is the case the comment was actually protecting. ## Testing - Java: **33 passing** (8 existing + 25 LSM) against a scripted `com.sun.net.httpserver.HttpServer` — no new test dependency. Wire assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`; checkpoint tests cover convergence, not piling onto a latched bucket, 421 restart-from-flush, 429 retry-in-place, terminal-status propagation, reissue exhaustion, the exact wire-request count against the retry budget, and five malformed stats payloads. - Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm run tsc`, `npm run lint`, `npm run docs` all clean. - Python: `ruff format --check` and `ruff check` clean. - Java formatting: `./mvnw -pl lancedb-core spotless:apply` and `spotless:check` both clean under a JDK 11 toolchain. ## Note: spotless needs a pre-16 JDK `./mvnw spotless:apply` fails on JDK 16+ with `JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7, pinned at `java/pom.xml:34`, predates JDK 16's compiler API change. **This is pre-existing** and reproduces on a pristine `main` checkout. It is not a blocker, just a toolchain requirement. Spotless was run against these sources under JDK 11 and both `spotless:apply` and `spotless:check` pass on the whole module: ```shell JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply ``` Bumping the plugin so it works on modern JDKs is still worth doing, but separately from this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dan Rammer
GitHub
@@ -213,6 +213,39 @@ version of the table.
|
||||
|
||||
***
|
||||
|
||||
### checkpointLsm()
|
||||
|
||||
```ts
|
||||
abstract checkpointLsm(): Promise<void>
|
||||
```
|
||||
|
||||
Converge this table's LSM write path into its base table.
|
||||
|
||||
Seals once, then triggers compaction and polls until the L0 that existed
|
||||
at the start is gone. The target set is fixed at the start, so
|
||||
generations created *during* the checkpoint are ignored — that is what
|
||||
lets it terminate under write load, and what makes it best-effort: it
|
||||
converges the fresh tier as of some instant. Idempotent, abandonable at
|
||||
any point, and safe to run on a cadence.
|
||||
|
||||
There is no liveness bound — the compactor pool is shared across tables,
|
||||
so a checkpoint queued behind unrelated work looks exactly like one that
|
||||
is merging. The caller owns the deadline.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`void`>
|
||||
|
||||
#### Example
|
||||
|
||||
```ts
|
||||
const before = await table.getLsmStats();
|
||||
await table.checkpointLsm();
|
||||
const after = await table.getLsmStats();
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### close()
|
||||
|
||||
```ts
|
||||
@@ -250,6 +283,24 @@ It is a no-op when no writers are cached.
|
||||
|
||||
***
|
||||
|
||||
### compactLsm()
|
||||
|
||||
```ts
|
||||
abstract compactLsm(): Promise<void>
|
||||
```
|
||||
|
||||
Trigger a background L0 → base compaction pass per bucket.
|
||||
|
||||
Returns once the passes are *dispatched*, not once they finish — watch
|
||||
[Table#getLsmStats](Table.md#getlsmstats) for progress, or use
|
||||
[Table#checkpointLsm](Table.md#checkpointlsm) to wait for convergence.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`void`>
|
||||
|
||||
***
|
||||
|
||||
### countRows()
|
||||
|
||||
```ts
|
||||
@@ -448,6 +499,48 @@ Drop an index from the table.
|
||||
|
||||
***
|
||||
|
||||
### flushLsm()
|
||||
|
||||
```ts
|
||||
abstract flushLsm(): Promise<void>
|
||||
```
|
||||
|
||||
Seal every bucket's active memtable into a new L0 generation.
|
||||
|
||||
Returns once the seal is committed. Sealing an empty memtable is a no-op,
|
||||
so this is safe to call repeatedly.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`void`>
|
||||
|
||||
***
|
||||
|
||||
### getLsmStats()
|
||||
|
||||
```ts
|
||||
abstract getLsmStats(includeGenerationRows?): Promise<undefined | LsmStats>
|
||||
```
|
||||
|
||||
Read live per-bucket LSM state.
|
||||
|
||||
Answers "how far behind is my fresh tier", "which bucket is hot", and
|
||||
"why is my fresh-tier vector search brute-force". Mutates no table state.
|
||||
|
||||
Resolves to `undefined` only when the LSM write path is not enabled.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **includeGenerationRows?**: `boolean`
|
||||
Also count rows per L0 generation.
|
||||
Off by default because each count opens an uncached Lance dataset.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`undefined` \| [`LsmStats`](../interfaces/LsmStats.md)>
|
||||
|
||||
***
|
||||
|
||||
### getLsmWriteSpec()
|
||||
|
||||
```ts
|
||||
|
||||
@@ -58,6 +58,7 @@
|
||||
- [BranchDiff](interfaces/BranchDiff.md)
|
||||
- [BranchIndexSummary](interfaces/BranchIndexSummary.md)
|
||||
- [BranchRowCountSummary](interfaces/BranchRowCountSummary.md)
|
||||
- [BucketStats](interfaces/BucketStats.md)
|
||||
- [ClientConfig](interfaces/ClientConfig.md)
|
||||
- [ColumnAlteration](interfaces/ColumnAlteration.md)
|
||||
- [ColumnOrdering](interfaces/ColumnOrdering.md)
|
||||
@@ -81,6 +82,7 @@
|
||||
- [FtsToken](interfaces/FtsToken.md)
|
||||
- [FullTextQuery](interfaces/FullTextQuery.md)
|
||||
- [FullTextSearchOptions](interfaces/FullTextSearchOptions.md)
|
||||
- [GenerationStats](interfaces/GenerationStats.md)
|
||||
- [HnswPqOptions](interfaces/HnswPqOptions.md)
|
||||
- [HnswSqOptions](interfaces/HnswSqOptions.md)
|
||||
- [IndexConfig](interfaces/IndexConfig.md)
|
||||
@@ -94,7 +96,9 @@
|
||||
- [JobInfo](interfaces/JobInfo.md)
|
||||
- [ListNamespacesOptions](interfaces/ListNamespacesOptions.md)
|
||||
- [ListNamespacesResponse](interfaces/ListNamespacesResponse.md)
|
||||
- [LsmStats](interfaces/LsmStats.md)
|
||||
- [LsmWriteSpec](interfaces/LsmWriteSpec.md)
|
||||
- [MemtableStats](interfaces/MemtableStats.md)
|
||||
- [MergeBlocker](interfaces/MergeBlocker.md)
|
||||
- [MergeBranchResult](interfaces/MergeBranchResult.md)
|
||||
- [MergePreview](interfaces/MergePreview.md)
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BucketStats
|
||||
|
||||
# Interface: BucketStats
|
||||
|
||||
Live state of one bucket. A table is N buckets on one node; flattening to a
|
||||
single number hides the one hot bucket that is usually why someone opened
|
||||
this endpoint.
|
||||
|
||||
## Properties
|
||||
|
||||
### compacting
|
||||
|
||||
```ts
|
||||
compacting: boolean;
|
||||
```
|
||||
|
||||
Whether a pass owns this bucket's compaction latch right now. Says *a*
|
||||
driver is running, not *whose*, and the latch is held from dispatch —
|
||||
including while the pass queues for a pod-wide compactor permit. Read it
|
||||
as "do not pile on", never as "mine is progressing".
|
||||
|
||||
***
|
||||
|
||||
### currentGeneration
|
||||
|
||||
```ts
|
||||
currentGeneration: number;
|
||||
```
|
||||
|
||||
The generation the active memtable will become.
|
||||
|
||||
***
|
||||
|
||||
### generations
|
||||
|
||||
```ts
|
||||
generations: GenerationStats[];
|
||||
```
|
||||
|
||||
Flushed L0 generations not yet merged into the base table.
|
||||
|
||||
***
|
||||
|
||||
### manifestVersion
|
||||
|
||||
```ts
|
||||
manifestVersion: number;
|
||||
```
|
||||
|
||||
Version of the shard manifest these numbers were read from.
|
||||
|
||||
***
|
||||
|
||||
### memtables?
|
||||
|
||||
```ts
|
||||
optional memtables: MemtableStats[];
|
||||
```
|
||||
|
||||
Oldest first, active last. Absent for a `"Sealed"` bucket, whose
|
||||
in-memory state is torn down.
|
||||
|
||||
***
|
||||
|
||||
### replayAfterWalEntryPosition
|
||||
|
||||
```ts
|
||||
replayAfterWalEntryPosition: number;
|
||||
```
|
||||
|
||||
WAL position replay resumes from.
|
||||
|
||||
***
|
||||
|
||||
### shardId
|
||||
|
||||
```ts
|
||||
shardId: string;
|
||||
```
|
||||
|
||||
The shard this bucket writes.
|
||||
|
||||
***
|
||||
|
||||
### status
|
||||
|
||||
```ts
|
||||
status: string;
|
||||
```
|
||||
|
||||
`"Active"` or `"Sealed"` (drop-table 2PC in flight).
|
||||
|
||||
***
|
||||
|
||||
### walEntryPositionLastSeen
|
||||
|
||||
```ts
|
||||
walEntryPositionLastSeen: number;
|
||||
```
|
||||
|
||||
Highest WAL position the writer has seen. The difference against
|
||||
`replayAfterWalEntryPosition` is the WAL lag.
|
||||
|
||||
***
|
||||
|
||||
### writerEpoch
|
||||
|
||||
```ts
|
||||
writerEpoch: number;
|
||||
```
|
||||
|
||||
Epoch of the writer that currently owns the shard.
|
||||
@@ -0,0 +1,40 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / GenerationStats
|
||||
|
||||
# Interface: GenerationStats
|
||||
|
||||
One flushed L0 generation.
|
||||
|
||||
## Properties
|
||||
|
||||
### bytes
|
||||
|
||||
```ts
|
||||
bytes: number;
|
||||
```
|
||||
|
||||
On-disk size of the generation.
|
||||
|
||||
***
|
||||
|
||||
### generation
|
||||
|
||||
```ts
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation number. Increases as memtables are sealed into L0.
|
||||
|
||||
***
|
||||
|
||||
### rows?
|
||||
|
||||
```ts
|
||||
optional rows: number;
|
||||
```
|
||||
|
||||
Present only when `includeGenerationRows` was requested. Off by default
|
||||
because each count opens an uncached Lance dataset.
|
||||
@@ -0,0 +1,22 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / LsmStats
|
||||
|
||||
# Interface: LsmStats
|
||||
|
||||
Live per-bucket LSM state, as returned by `Table#getLsmStats`.
|
||||
|
||||
Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are
|
||||
the caller's to compute.
|
||||
|
||||
## Properties
|
||||
|
||||
### buckets
|
||||
|
||||
```ts
|
||||
buckets: BucketStats[];
|
||||
```
|
||||
|
||||
One entry per bucket backing this table.
|
||||
@@ -0,0 +1,60 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / MemtableStats
|
||||
|
||||
# Interface: MemtableStats
|
||||
|
||||
One in-memory memtable.
|
||||
|
||||
## Properties
|
||||
|
||||
### batches
|
||||
|
||||
```ts
|
||||
batches: number;
|
||||
```
|
||||
|
||||
Record batches currently buffered.
|
||||
|
||||
***
|
||||
|
||||
### bytes
|
||||
|
||||
```ts
|
||||
bytes: number;
|
||||
```
|
||||
|
||||
Estimated in-memory size.
|
||||
|
||||
***
|
||||
|
||||
### generation
|
||||
|
||||
```ts
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation this memtable will become once sealed.
|
||||
|
||||
***
|
||||
|
||||
### indexes
|
||||
|
||||
```ts
|
||||
indexes: string[];
|
||||
```
|
||||
|
||||
Names of the indexes this memtable carries. An absent name is the whole
|
||||
answer to "why is my fresh-tier search on that column brute-force".
|
||||
|
||||
***
|
||||
|
||||
### rows
|
||||
|
||||
```ts
|
||||
rows: number;
|
||||
```
|
||||
|
||||
Rows currently buffered.
|
||||
@@ -52,6 +52,8 @@ listing a storage directory.
|
||||
|
||||
::: lancedb.table.Branches
|
||||
|
||||
::: lancedb.LsmWriteSpec
|
||||
|
||||
## Expressions
|
||||
|
||||
Type-safe expression builder for filters and projections. Use these instead
|
||||
|
||||
@@ -29,6 +29,48 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.build();
|
||||
```
|
||||
|
||||
## MemWAL LSM write path
|
||||
|
||||
Most table operations reach LanceDB through the `LanceNamespace` above, which is
|
||||
generated from the Lance Namespace specification. The MemWAL LSM routes are not part
|
||||
of that specification, so they are issued through a separate client:
|
||||
|
||||
```java
|
||||
import com.lancedb.LanceDbRestClient;
|
||||
import com.lancedb.LanceDbTableLsm;
|
||||
import com.lancedb.LsmWriteSpec;
|
||||
|
||||
LanceDbRestClient client = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.apiKey("your_lancedb_cloud_api_key")
|
||||
.database("your_database_name")
|
||||
.buildRestClient();
|
||||
|
||||
LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");
|
||||
|
||||
// Route future merge_insert upserts through the MemWAL, hash-bucketed by `id`.
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));
|
||||
|
||||
// ... merge_insert traffic ...
|
||||
|
||||
// Converge the fresh tier into the base table.
|
||||
lsm.checkpointLsm();
|
||||
|
||||
// Inspect live per-bucket state.
|
||||
lsm.getLsmStats().ifPresent(stats -> stats.buckets().forEach(bucket ->
|
||||
System.out.println(bucket.shardId() + ": " + bucket.generations().size() + " L0 generations")));
|
||||
|
||||
client.close();
|
||||
```
|
||||
|
||||
`maintainedIndexes` is tri-state, and the null default is the opposite of what a Java
|
||||
reader usually expects:
|
||||
|
||||
| Value | Meaning |
|
||||
| --- | --- |
|
||||
| unset (null) | Maintain **every** index the MemWAL can, resolved on install |
|
||||
| `Collections.emptyList()` | Maintain **none** |
|
||||
| `Arrays.asList("id_idx")` | Maintain exactly those |
|
||||
|
||||
## Development
|
||||
|
||||
Build:
|
||||
|
||||
@@ -33,6 +33,20 @@
|
||||
<artifactId>arrow-memory-netty</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- Transport for the LanceDB routes outside the Lance Namespace spec.
|
||||
Versions match what lance-namespace-apache-client resolves to. -->
|
||||
<dependency>
|
||||
<groupId>org.apache.httpcomponents.client5</groupId>
|
||||
<artifactId>httpclient5</artifactId>
|
||||
<version>5.2.1</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.fasterxml.jackson.core</groupId>
|
||||
<artifactId>jackson-databind</artifactId>
|
||||
<version>2.17.1</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter</artifactId>
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/**
|
||||
* Live state of one bucket. A table is N buckets on one node; flattening to a single number hides
|
||||
* the one hot bucket that is usually why someone opened this endpoint.
|
||||
*/
|
||||
public class BucketStats {
|
||||
private static final String CONTEXT = "bucket stats";
|
||||
|
||||
private final String shardId;
|
||||
private final String status;
|
||||
private final long writerEpoch;
|
||||
private final long manifestVersion;
|
||||
private final long currentGeneration;
|
||||
private final long replayAfterWalEntryPosition;
|
||||
private final long walEntryPositionLastSeen;
|
||||
private final List<GenerationStats> generations;
|
||||
private final boolean compacting;
|
||||
private final List<MemtableStats> memtables;
|
||||
|
||||
BucketStats(
|
||||
String shardId,
|
||||
String status,
|
||||
long writerEpoch,
|
||||
long manifestVersion,
|
||||
long currentGeneration,
|
||||
long replayAfterWalEntryPosition,
|
||||
long walEntryPositionLastSeen,
|
||||
List<GenerationStats> generations,
|
||||
boolean compacting,
|
||||
List<MemtableStats> memtables) {
|
||||
this.shardId = shardId;
|
||||
this.status = status;
|
||||
this.writerEpoch = writerEpoch;
|
||||
this.manifestVersion = manifestVersion;
|
||||
this.currentGeneration = currentGeneration;
|
||||
this.replayAfterWalEntryPosition = replayAfterWalEntryPosition;
|
||||
this.walEntryPositionLastSeen = walEntryPositionLastSeen;
|
||||
this.generations = Collections.unmodifiableList(generations);
|
||||
this.compacting = compacting;
|
||||
this.memtables = memtables == null ? null : Collections.unmodifiableList(memtables);
|
||||
}
|
||||
|
||||
/** The shard this bucket writes. */
|
||||
public String shardId() {
|
||||
return shardId;
|
||||
}
|
||||
|
||||
/** {@code "Active"} or {@code "Sealed"} (drop-table 2PC in flight). */
|
||||
public String status() {
|
||||
return status;
|
||||
}
|
||||
|
||||
/** Epoch of the writer that currently owns the shard. */
|
||||
public long writerEpoch() {
|
||||
return writerEpoch;
|
||||
}
|
||||
|
||||
/** Version of the shard manifest these numbers were read from. */
|
||||
public long manifestVersion() {
|
||||
return manifestVersion;
|
||||
}
|
||||
|
||||
/** The generation the active memtable will become. */
|
||||
public long currentGeneration() {
|
||||
return currentGeneration;
|
||||
}
|
||||
|
||||
/** WAL position replay resumes from. */
|
||||
public long replayAfterWalEntryPosition() {
|
||||
return replayAfterWalEntryPosition;
|
||||
}
|
||||
|
||||
/**
|
||||
* Highest WAL position the writer has seen. The difference against {@link
|
||||
* #replayAfterWalEntryPosition()} is the WAL lag.
|
||||
*/
|
||||
public long walEntryPositionLastSeen() {
|
||||
return walEntryPositionLastSeen;
|
||||
}
|
||||
|
||||
/** Flushed L0 generations not yet merged into the base table. */
|
||||
public List<GenerationStats> generations() {
|
||||
return generations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a pass owns this bucket's compaction latch right now. Says <em>a</em> driver is
|
||||
* running, not <em>whose</em>, and the latch is held from dispatch — including while the pass
|
||||
* queues for a pod-wide compactor permit. Read it as "do not pile on", never as "mine is
|
||||
* progressing".
|
||||
*/
|
||||
public boolean compacting() {
|
||||
return compacting;
|
||||
}
|
||||
|
||||
/** Oldest first, active last. Empty for a {@code "Sealed"} bucket, whose state is torn down. */
|
||||
public Optional<List<MemtableStats>> memtables() {
|
||||
return Optional.ofNullable(memtables);
|
||||
}
|
||||
|
||||
/** The newest flushed generation, or empty when L0 is empty. */
|
||||
OptionalLong newestGeneration() {
|
||||
OptionalLong newest = OptionalLong.empty();
|
||||
for (GenerationStats generation : generations) {
|
||||
if (!newest.isPresent() || generation.generation() > newest.getAsLong()) {
|
||||
newest = OptionalLong.of(generation.generation());
|
||||
}
|
||||
}
|
||||
return newest;
|
||||
}
|
||||
|
||||
/**
|
||||
* How many generations at or below {@code target} are still in L0.
|
||||
*
|
||||
* <p>A count, not a boolean: one pass drains a bounded prefix rather than the whole target set,
|
||||
* so a boolean would read as "no progress" for every pass but the last. Compaction drains
|
||||
* oldest-first, so this decreases monotonically.
|
||||
*/
|
||||
long outstandingGenerations(long target) {
|
||||
long count = 0;
|
||||
for (GenerationStats generation : generations) {
|
||||
if (generation.generation() <= target) {
|
||||
count++;
|
||||
}
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
static BucketStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
List<GenerationStats> generations = new ArrayList<GenerationStats>();
|
||||
for (JsonNode generation : JsonFields.requiredArray(node, "generations", CONTEXT)) {
|
||||
generations.add(GenerationStats.fromJson(generation));
|
||||
}
|
||||
|
||||
JsonNode memtablesNode = JsonFields.optionalArray(node, "memtables", CONTEXT);
|
||||
List<MemtableStats> memtables = null;
|
||||
if (memtablesNode != null) {
|
||||
memtables = new ArrayList<MemtableStats>();
|
||||
for (JsonNode memtable : memtablesNode) {
|
||||
memtables.add(MemtableStats.fromJson(memtable));
|
||||
}
|
||||
}
|
||||
|
||||
return new BucketStats(
|
||||
JsonFields.requiredText(node, "shard_id", CONTEXT),
|
||||
JsonFields.requiredText(node, "status", CONTEXT),
|
||||
JsonFields.requiredLong(node, "writer_epoch", CONTEXT),
|
||||
JsonFields.requiredLong(node, "manifest_version", CONTEXT),
|
||||
JsonFields.requiredLong(node, "current_generation", CONTEXT),
|
||||
JsonFields.requiredLong(node, "replay_after_wal_entry_position", CONTEXT),
|
||||
JsonFields.requiredLong(node, "wal_entry_position_last_seen", CONTEXT),
|
||||
generations,
|
||||
JsonFields.requiredBoolean(node, "compacting", CONTEXT),
|
||||
memtables);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "BucketStats{shardId="
|
||||
+ shardId
|
||||
+ ", status="
|
||||
+ status
|
||||
+ ", currentGeneration="
|
||||
+ currentGeneration
|
||||
+ ", generations="
|
||||
+ generations
|
||||
+ ", compacting="
|
||||
+ compacting
|
||||
+ "}";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/** One flushed L0 generation. */
|
||||
public class GenerationStats {
|
||||
private static final String CONTEXT = "generation stats";
|
||||
|
||||
private final long generation;
|
||||
private final long bytes;
|
||||
private final Long rows;
|
||||
|
||||
GenerationStats(long generation, long bytes, Long rows) {
|
||||
this.generation = generation;
|
||||
this.bytes = bytes;
|
||||
this.rows = rows;
|
||||
}
|
||||
|
||||
/** The generation number. Increases as memtables are sealed into L0. */
|
||||
public long generation() {
|
||||
return generation;
|
||||
}
|
||||
|
||||
/** On-disk size of the generation. */
|
||||
public long bytes() {
|
||||
return bytes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Rows in this generation, present only when {@code includeGenerationRows} was requested. Off by
|
||||
* default because each count opens an uncached Lance dataset.
|
||||
*/
|
||||
public OptionalLong rows() {
|
||||
return rows == null ? OptionalLong.empty() : OptionalLong.of(rows);
|
||||
}
|
||||
|
||||
static GenerationStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
return new GenerationStats(
|
||||
JsonFields.requiredLong(node, "generation", CONTEXT),
|
||||
JsonFields.requiredLong(node, "bytes", CONTEXT),
|
||||
JsonFields.optionalLong(node, "rows", CONTEXT));
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "GenerationStats{generation=" + generation + ", bytes=" + bytes + ", rows=" + rows + "}";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
/**
|
||||
* Strict readers for decoding LanceDB JSON responses.
|
||||
*
|
||||
* <p>Every reader fails closed: a missing, null, or wrong-typed field throws rather than
|
||||
* defaulting. That mirrors the serde decoding the Rust client applies to the same payloads in
|
||||
* {@code rust/lancedb/src/table/lsm_stats.rs}, where a required field has no default and a
|
||||
* malformed response is an error rather than a zero.
|
||||
*
|
||||
* <p>The alternative — Jackson's {@code path()}, which yields a missing node that reads as an empty
|
||||
* array or a zero — is unsafe here because {@link LanceDbTableLsm#checkpointLsm()} decides
|
||||
* convergence from these numbers. A defaulted {@code generations} array is indistinguishable from a
|
||||
* drained one, so a malformed response would report a checkpoint that never happened.
|
||||
*/
|
||||
final class JsonFields {
|
||||
private JsonFields() {}
|
||||
|
||||
/** The node itself, once confirmed to be a JSON object. */
|
||||
static JsonNode requiredObject(JsonNode node, String context) {
|
||||
if (node == null || !node.isObject()) {
|
||||
throw new IllegalStateException(context + " is not a JSON object: " + node);
|
||||
}
|
||||
return node;
|
||||
}
|
||||
|
||||
static String requiredText(JsonNode owner, String field, String context) {
|
||||
JsonNode value = required(owner, field, context);
|
||||
if (!value.isTextual()) {
|
||||
throw new IllegalStateException(fieldIs(context, field, "a string", value));
|
||||
}
|
||||
return value.asText();
|
||||
}
|
||||
|
||||
static long requiredLong(JsonNode owner, String field, String context) {
|
||||
JsonNode value = required(owner, field, context);
|
||||
if (!value.isIntegralNumber()) {
|
||||
throw new IllegalStateException(fieldIs(context, field, "an integer", value));
|
||||
}
|
||||
return value.asLong();
|
||||
}
|
||||
|
||||
static boolean requiredBoolean(JsonNode owner, String field, String context) {
|
||||
JsonNode value = required(owner, field, context);
|
||||
if (!value.isBoolean()) {
|
||||
throw new IllegalStateException(fieldIs(context, field, "a boolean", value));
|
||||
}
|
||||
return value.asBoolean();
|
||||
}
|
||||
|
||||
static JsonNode requiredArray(JsonNode owner, String field, String context) {
|
||||
JsonNode value = required(owner, field, context);
|
||||
if (!value.isArray()) {
|
||||
throw new IllegalStateException(fieldIs(context, field, "an array", value));
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/** Null when the field is absent or JSON null, mirroring a serde {@code Option}. */
|
||||
static Long optionalLong(JsonNode owner, String field, String context) {
|
||||
JsonNode value = owner.get(field);
|
||||
if (value == null || value.isNull()) {
|
||||
return null;
|
||||
}
|
||||
if (!value.isIntegralNumber()) {
|
||||
throw new IllegalStateException(fieldIs(context, field, "an integer", value));
|
||||
}
|
||||
return value.asLong();
|
||||
}
|
||||
|
||||
/** Null when the field is absent or JSON null, mirroring a serde {@code Option}. */
|
||||
static JsonNode optionalArray(JsonNode owner, String field, String context) {
|
||||
JsonNode value = owner.get(field);
|
||||
if (value == null || value.isNull()) {
|
||||
return null;
|
||||
}
|
||||
if (!value.isArray()) {
|
||||
throw new IllegalStateException(fieldIs(context, field, "an array", value));
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
private static JsonNode required(JsonNode owner, String field, String context) {
|
||||
JsonNode value = owner.get(field);
|
||||
if (value == null || value.isNull()) {
|
||||
throw new IllegalStateException(context + " is missing required field '" + field + "'");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
private static String fieldIs(String context, String field, String expected, JsonNode value) {
|
||||
return context + " field '" + field + "' is not " + expected + ": " + value;
|
||||
}
|
||||
}
|
||||
@@ -136,29 +136,48 @@ public class LanceDbNamespaceClientBuilder {
|
||||
* @throws IllegalStateException if required parameters are missing
|
||||
*/
|
||||
public LanceNamespace build() {
|
||||
// Validate required fields
|
||||
validate();
|
||||
|
||||
// Build configuration map
|
||||
Map<String, String> config = new HashMap<>(additionalConfig);
|
||||
config.put("header.x-lancedb-database", database);
|
||||
config.put("header.x-api-key", apiKey);
|
||||
config.put("uri", resolveUri());
|
||||
|
||||
return LanceNamespace.connect("rest", config, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a {@link LanceDbRestClient} for the same endpoint.
|
||||
*
|
||||
* <p>Needed only for LanceDB routes that the Lance Namespace specification does not cover — the
|
||||
* MemWAL LSM write path, reached through {@link LanceDbTableLsm}. Every other table operation
|
||||
* belongs on the {@link LanceNamespace} from {@link #build()}.
|
||||
*
|
||||
* <p>The returned client owns an HTTP connection pool; close it when you are done with it.
|
||||
*
|
||||
* @return A configured LanceDbRestClient
|
||||
* @throws IllegalStateException if required parameters are missing
|
||||
*/
|
||||
public LanceDbRestClient buildRestClient() {
|
||||
validate();
|
||||
return new LanceDbRestClient(resolveUri(), apiKey, database);
|
||||
}
|
||||
|
||||
private void validate() {
|
||||
if (apiKey == null) {
|
||||
throw new IllegalStateException("API key is required");
|
||||
}
|
||||
if (database == null) {
|
||||
throw new IllegalStateException("Database is required");
|
||||
}
|
||||
}
|
||||
|
||||
// Build configuration map
|
||||
Map<String, String> config = new HashMap<>(additionalConfig);
|
||||
config.put("header.x-lancedb-database", database);
|
||||
config.put("header.x-api-key", apiKey);
|
||||
|
||||
// Determine base URL
|
||||
String uri;
|
||||
/** The custom endpoint when set, else the LanceDB Cloud URL for this database and region. */
|
||||
private String resolveUri() {
|
||||
if (endpoint.isPresent()) {
|
||||
uri = endpoint.get();
|
||||
} else {
|
||||
String effectiveRegion = region.orElse(DEFAULT_REGION);
|
||||
uri = String.format(CLOUD_URL_PATTERN, database, effectiveRegion);
|
||||
return endpoint.get();
|
||||
}
|
||||
config.put("uri", uri);
|
||||
|
||||
return LanceNamespace.connect("rest", config, null);
|
||||
return String.format(CLOUD_URL_PATTERN, database, region.orElse(DEFAULT_REGION));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.apache.hc.client5.http.classic.methods.HttpPost;
|
||||
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
|
||||
import org.apache.hc.client5.http.impl.classic.HttpClients;
|
||||
import org.apache.hc.core5.http.ContentType;
|
||||
import org.apache.hc.core5.http.io.entity.EntityUtils;
|
||||
import org.apache.hc.core5.http.io.entity.StringEntity;
|
||||
|
||||
import java.io.Closeable;
|
||||
import java.io.IOException;
|
||||
import java.io.UncheckedIOException;
|
||||
|
||||
/**
|
||||
* Minimal HTTP client for LanceDB Cloud and Enterprise routes that the Lance Namespace
|
||||
* specification does not cover.
|
||||
*
|
||||
* <p>Most table operations reach LanceDB through {@link org.lance.namespace.LanceNamespace}, which
|
||||
* is generated from the namespace spec. A handful of routes — the MemWAL LSM write path in
|
||||
* particular — are served by the same endpoint but are not part of that spec, so they are issued
|
||||
* directly here. See {@link LanceDbTableLsm}.
|
||||
*
|
||||
* <p>Obtain one from {@link LanceDbNamespaceClientBuilder#buildRestClient()}.
|
||||
*/
|
||||
public class LanceDbRestClient implements Closeable {
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
|
||||
private final String baseUri;
|
||||
private final String apiKey;
|
||||
private final String database;
|
||||
private final CloseableHttpClient http;
|
||||
|
||||
LanceDbRestClient(String baseUri, String apiKey, String database) {
|
||||
this.baseUri = baseUri.endsWith("/") ? baseUri.substring(0, baseUri.length() - 1) : baseUri;
|
||||
this.apiKey = apiKey;
|
||||
this.database = database;
|
||||
// Automatic retries off, deliberately. The default strategy retries 429 and 503 —
|
||||
// exactly the two statuses LanceDbTableLsm.checkpointLsm() acts on — which would
|
||||
// silently double its explicit retry budget and would also retry compact_lsm in
|
||||
// place, where the loop is designed to fall through to a fresh stats poll instead.
|
||||
// The checkpoint loop owns the 421/429/503 transitions; the transport must not.
|
||||
this.http = HttpClients.custom().disableAutomaticRetries().build();
|
||||
}
|
||||
|
||||
/**
|
||||
* POST {@code path}, sending {@code body} as JSON when it is non-null.
|
||||
*
|
||||
* @param path Absolute request path, beginning with {@code /}.
|
||||
* @param body Object to serialize as the request body, or null to send no body.
|
||||
* @return The parsed response body, or null when the response carried no content.
|
||||
* @throws HttpException if the server returned a non-2xx status.
|
||||
*/
|
||||
public JsonNode post(String path, Object body) {
|
||||
HttpPost request = new HttpPost(baseUri + path);
|
||||
request.setHeader("x-api-key", apiKey);
|
||||
request.setHeader("x-lancedb-database", database);
|
||||
try {
|
||||
if (body != null) {
|
||||
request.setEntity(
|
||||
new StringEntity(MAPPER.writeValueAsString(body), ContentType.APPLICATION_JSON));
|
||||
}
|
||||
return http.execute(
|
||||
request,
|
||||
response -> {
|
||||
String text =
|
||||
response.getEntity() == null ? "" : EntityUtils.toString(response.getEntity());
|
||||
int status = response.getCode();
|
||||
if (status < 200 || status >= 300) {
|
||||
throw new HttpException(status, "LanceDB request to " + path + " failed: " + text);
|
||||
}
|
||||
return text.isEmpty() ? null : MAPPER.readTree(text);
|
||||
});
|
||||
} catch (IOException e) {
|
||||
throw new UncheckedIOException("LanceDB request to " + path + " failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() throws IOException {
|
||||
http.close();
|
||||
}
|
||||
|
||||
/**
|
||||
* A non-2xx response.
|
||||
*
|
||||
* <p>The status is exposed because callers act on it: {@link LanceDbTableLsm#checkpointLsm()}
|
||||
* treats 429 and 503 as retryable and 421 as a lost node claim.
|
||||
*/
|
||||
public static class HttpException extends RuntimeException {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private final int statusCode;
|
||||
|
||||
public HttpException(int statusCode, String message) {
|
||||
super(message);
|
||||
this.statusCode = statusCode;
|
||||
}
|
||||
|
||||
/** The HTTP status the failed response carried. */
|
||||
public int statusCode() {
|
||||
return statusCode;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,394 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.OptionalLong;
|
||||
|
||||
/**
|
||||
* The MemWAL LSM write path for one LanceDB Cloud or Enterprise table.
|
||||
*
|
||||
* <p>Installing an {@link LsmWriteSpec} routes {@code mergeInsert} upserts through Lance's MemWAL —
|
||||
* an LSM-style append — instead of the standard merge path. Rows land in an in-memory memtable,
|
||||
* seal into L0 generations, and are merged into the base table by compaction.
|
||||
*
|
||||
* <p>These routes are not part of the Lance Namespace specification, so they are issued directly
|
||||
* rather than through {@link org.lance.namespace.LanceNamespace}.
|
||||
*
|
||||
* <pre>{@code
|
||||
* LanceDbRestClient client = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
* .apiKey("your_lancedb_cloud_api_key")
|
||||
* .database("your_database_name")
|
||||
* .buildRestClient();
|
||||
*
|
||||
* LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");
|
||||
* lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));
|
||||
* // ... merge_insert traffic ...
|
||||
* lsm.checkpointLsm();
|
||||
* }</pre>
|
||||
*/
|
||||
public class LanceDbTableLsm {
|
||||
|
||||
/**
|
||||
* Interval between {@code get_lsm_stats} polls during a checkpoint. One interval is roughly one
|
||||
* compaction pass, the granularity at which the answer can change.
|
||||
*/
|
||||
private static final long POLL_INTERVAL_MS = 5_000L;
|
||||
|
||||
/**
|
||||
* Cap on re-issues from {@code flushLsm} after a 421, so a crash-looping node cannot turn flush →
|
||||
* compact → 421 → flush into a spin.
|
||||
*
|
||||
* <p>Deliberately not shared with {@link #MAX_RETRIES}: a claim that keeps evaporating is a
|
||||
* broken node, while contention is routine and wants a real budget.
|
||||
*/
|
||||
private static final int MAX_REISSUES = 3;
|
||||
|
||||
/**
|
||||
* Retryable faults tolerated on a <em>single</em> request, reset on every success — scattered
|
||||
* contention across a long checkpoint must not accumulate toward a cap.
|
||||
*/
|
||||
private static final int MAX_RETRIES = 8;
|
||||
|
||||
private static final long RETRY_BACKOFF_BASE_MS = 100L;
|
||||
private static final long RETRY_BACKOFF_MAX_MS = 5_000L;
|
||||
|
||||
private final LanceDbRestClient client;
|
||||
private final String tableIdentifier;
|
||||
|
||||
/**
|
||||
* Bind the LSM routes for one table.
|
||||
*
|
||||
* @param client Transport for the LanceDB endpoint.
|
||||
* @param tableIdentifier The table's full identifier, {@code $}-delimited when it sits inside a
|
||||
* namespace, such as {@code analytics$events}.
|
||||
*/
|
||||
public LanceDbTableLsm(LanceDbRestClient client, String tableIdentifier) {
|
||||
if (client == null) {
|
||||
throw new IllegalArgumentException("Client cannot be null");
|
||||
}
|
||||
if (tableIdentifier == null || tableIdentifier.trim().isEmpty()) {
|
||||
throw new IllegalArgumentException("Table identifier cannot be null or empty");
|
||||
}
|
||||
this.client = client;
|
||||
this.tableIdentifier = tableIdentifier;
|
||||
}
|
||||
|
||||
/**
|
||||
* Install an {@link LsmWriteSpec} on this table, selecting the MemWAL LSM write path for future
|
||||
* {@code mergeInsert} calls.
|
||||
*
|
||||
* <p>All variants require the table to have an unenforced primary key; bucket sharding
|
||||
* additionally requires it to be the single column being bucketed.
|
||||
*/
|
||||
public void setLsmWriteSpec(LsmWriteSpec spec) {
|
||||
if (spec == null) {
|
||||
throw new IllegalArgumentException("Spec cannot be null");
|
||||
}
|
||||
client.post(route("set_lsm_write_spec"), spec.toRequestBody());
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove the {@link LsmWriteSpec} from this table, reverting to the standard {@code mergeInsert}
|
||||
* write path.
|
||||
*
|
||||
* <p>Errors if no spec is currently set.
|
||||
*/
|
||||
public void unsetLsmWriteSpec() {
|
||||
client.post(route("unset_lsm_write_spec"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the {@link LsmWriteSpec} currently installed on this table.
|
||||
*
|
||||
* <p>Empty when the LSM write path is not enabled. The returned spec mirrors what was installed,
|
||||
* except that {@link LsmWriteSpec#maintainedIndexes()} always reports the concrete list resolved
|
||||
* when the spec was set — a null selection never round-trips.
|
||||
*/
|
||||
public Optional<LsmWriteSpec> getLsmWriteSpec() {
|
||||
JsonNode response = client.post(route("get_lsm_write_spec"), null);
|
||||
if (response == null || !response.hasNonNull("lsm_write_spec")) {
|
||||
return Optional.empty();
|
||||
}
|
||||
return Optional.of(LsmWriteSpec.fromJson(response.get("lsm_write_spec")));
|
||||
}
|
||||
|
||||
/**
|
||||
* Seal every bucket's active memtable into a new L0 generation.
|
||||
*
|
||||
* <p>Returns once the seal is committed. Sealing an empty memtable is a no-op, so this is safe to
|
||||
* call repeatedly.
|
||||
*/
|
||||
public void flushLsm() {
|
||||
client.post(route("flush_lsm"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger a background L0 → base compaction pass per bucket.
|
||||
*
|
||||
* <p>Returns once the passes are <em>dispatched</em>, not once they finish — watch {@link
|
||||
* #getLsmStats}, or use {@link #checkpointLsm} to wait for convergence.
|
||||
*/
|
||||
public void compactLsm() {
|
||||
client.post(route("compact_lsm"), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read live per-bucket LSM state.
|
||||
*
|
||||
* <p>Answers "how far behind is my fresh tier", "which bucket is hot", and "why is my fresh-tier
|
||||
* vector search brute-force". Mutates no table state.
|
||||
*
|
||||
* <p>Empty only when the LSM write path is not enabled — that is, when the server sends an absent
|
||||
* or null {@code lsm_stats}. A stats object that is present is decoded strictly, and a malformed
|
||||
* one throws rather than decoding to something empty, because {@link #checkpointLsm} reads
|
||||
* convergence out of these numbers and cannot tell a defaulted array from a drained one.
|
||||
*
|
||||
* @param includeGenerationRows Also count rows per L0 generation. Off by default because each
|
||||
* count opens an uncached Lance dataset.
|
||||
* @throws IllegalStateException if the response is absent or does not decode.
|
||||
*/
|
||||
public Optional<LsmStats> getLsmStats(boolean includeGenerationRows) {
|
||||
Map<String, Object> body = new LinkedHashMap<String, Object>();
|
||||
body.put("include_generation_rows", includeGenerationRows);
|
||||
JsonNode response = client.post(route("get_lsm_stats"), body);
|
||||
if (response == null) {
|
||||
throw new IllegalStateException("get_lsm_stats returned an empty response body");
|
||||
}
|
||||
JsonNode stats = response.get("lsm_stats");
|
||||
if (stats == null || stats.isNull()) {
|
||||
return Optional.empty();
|
||||
}
|
||||
return Optional.of(LsmStats.fromJson(stats));
|
||||
}
|
||||
|
||||
/** Equivalent to {@code getLsmStats(false)}. */
|
||||
public Optional<LsmStats> getLsmStats() {
|
||||
return getLsmStats(false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Converge this table's LSM write path into its base table.
|
||||
*
|
||||
* <p>Seals once, fixes a target watermark from the resulting L0, then triggers compaction and
|
||||
* polls until that L0 is gone. The target set is fixed at the start, so generations created
|
||||
* <em>during</em> the checkpoint are ignored — that is what lets it terminate under write load,
|
||||
* and what makes it best-effort: it converges the fresh tier as of some instant. Idempotent,
|
||||
* abandonable at any point, safe on a cadence.
|
||||
*
|
||||
* <p>The loop runs here, not on the server: {@link #compactLsm} dispatches a pass and returns, so
|
||||
* nothing holds a socket and a client can vanish mid-operation with nothing to reconcile.
|
||||
* Completion is read from generation numbers in the shard manifest — durable state, unlike a
|
||||
* count in a compact response, which a concurrent write invalidates.
|
||||
*
|
||||
* <p>No liveness bound — the caller owns the deadline. The compactor pool is shared across
|
||||
* tables, so a checkpoint queued behind unrelated work looks exactly like one that is merging.
|
||||
*/
|
||||
public void checkpointLsm() {
|
||||
for (int reissue = 0; reissue <= MAX_REISSUES; reissue++) {
|
||||
// The seal turns everything written before this call into a generation, so the
|
||||
// watermark has to be read after it. Idempotent: sealing an empty memtable is a
|
||||
// no-op, so a re-issue does not churn empty generations.
|
||||
if (issueVoid(this::flushLsm)) {
|
||||
backoff(reissue);
|
||||
continue;
|
||||
}
|
||||
|
||||
Attempt<Optional<LsmStats>> stats = issue(() -> getLsmStats(false));
|
||||
if (stats.lostClaim) {
|
||||
backoff(reissue);
|
||||
continue;
|
||||
}
|
||||
if (!stats.value.isPresent()) {
|
||||
// Not WAL-backed; flushLsm would have errored first but for a race.
|
||||
return;
|
||||
}
|
||||
|
||||
Map<String, Long> targets = newestGenerations(stats.value.get());
|
||||
if (targets.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (drainToTargets(targets)) {
|
||||
return;
|
||||
}
|
||||
backoff(reissue);
|
||||
}
|
||||
throw new IllegalStateException(
|
||||
"checkpointLsm: the owning node kept losing its claim; re-issued from flush the maximum "
|
||||
+ "number of times");
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger and poll until no bucket holds a generation at or below its target.
|
||||
*
|
||||
* @return true when the drain finished, false when the table needs re-claiming from flush.
|
||||
*/
|
||||
private boolean drainToTargets(Map<String, Long> targets) {
|
||||
while (true) {
|
||||
Attempt<Optional<LsmStats>> stats = issue(() -> getLsmStats(false));
|
||||
if (stats.lostClaim) {
|
||||
return false;
|
||||
}
|
||||
if (!stats.value.isPresent()) {
|
||||
return true;
|
||||
}
|
||||
|
||||
// `compacting` is the bucket's compaction latch, held from dispatch until the pass
|
||||
// ends — including while it waits on a pod-wide permit. So it answers one question
|
||||
// only: do not pile on. Buckets with nothing outstanding are skipped, not counted
|
||||
// as idle.
|
||||
long outstanding = 0;
|
||||
boolean allCompacting = true;
|
||||
for (BucketStats bucket : stats.value.get().buckets()) {
|
||||
Long target = targets.get(bucket.shardId());
|
||||
if (target == null) {
|
||||
continue;
|
||||
}
|
||||
long remaining = bucket.outstandingGenerations(target);
|
||||
if (remaining > 0) {
|
||||
outstanding += remaining;
|
||||
allCompacting &= bucket.compacting();
|
||||
}
|
||||
}
|
||||
if (outstanding == 0) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!allCompacting) {
|
||||
try {
|
||||
compactLsm();
|
||||
} catch (LanceDbRestClient.HttpException e) {
|
||||
if (isLostClaim(e)) {
|
||||
return false;
|
||||
}
|
||||
if (!isRetryable(e)) {
|
||||
throw e;
|
||||
}
|
||||
// A 429 here means the server could latch no bucket at all, which the poll
|
||||
// above already handles. Not retried in place: the latch it would contend for
|
||||
// is the one doing the work, so fall through and re-read — POLL_INTERVAL_MS is
|
||||
// the backoff.
|
||||
}
|
||||
}
|
||||
sleep(POLL_INTERVAL_MS);
|
||||
}
|
||||
}
|
||||
|
||||
/** The newest generation held by each bucket, skipping buckets holding none. */
|
||||
private static Map<String, Long> newestGenerations(LsmStats stats) {
|
||||
Map<String, Long> targets = new HashMap<String, Long>();
|
||||
for (BucketStats bucket : stats.buckets()) {
|
||||
OptionalLong newest = bucket.newestGeneration();
|
||||
if (newest.isPresent()) {
|
||||
targets.put(bucket.shardId(), newest.getAsLong());
|
||||
}
|
||||
}
|
||||
return targets;
|
||||
}
|
||||
|
||||
/**
|
||||
* 429 (latch held, pool saturated, or the pod replaying its WAL) and 503 (a draining node, or a
|
||||
* proxy between here and it).
|
||||
*/
|
||||
private static boolean isRetryable(LanceDbRestClient.HttpException e) {
|
||||
return e.statusCode() == 429 || e.statusCode() == 503;
|
||||
}
|
||||
|
||||
/**
|
||||
* 421: the owning node holds no claim. Only {@code flush} re-claims and replays, so this cannot
|
||||
* be retried in place — the caller has to start over.
|
||||
*/
|
||||
private static boolean isLostClaim(LanceDbRestClient.HttpException e) {
|
||||
return e.statusCode() == 421;
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue one LSM request, retrying in place while the fault is retryable.
|
||||
*
|
||||
* <p>The two recoverable faults have separate budgets: contention clears on its own and retries
|
||||
* here against {@link #MAX_RETRIES}, while a 421 needs {@code flush} to re-claim, which only the
|
||||
* caller can drive.
|
||||
*
|
||||
* <p>An exhausted budget propagates the last error as itself rather than a synthesized one — "429
|
||||
* after nine tries" beats "checkpoint failed".
|
||||
*/
|
||||
private static <T> Attempt<T> issue(Call<T> call) {
|
||||
int retries = 0;
|
||||
while (true) {
|
||||
try {
|
||||
return new Attempt<T>(call.run(), false);
|
||||
} catch (LanceDbRestClient.HttpException e) {
|
||||
if (isLostClaim(e)) {
|
||||
return new Attempt<T>(null, true);
|
||||
}
|
||||
if (!isRetryable(e) || retries >= MAX_RETRIES) {
|
||||
throw e;
|
||||
}
|
||||
backoff(retries);
|
||||
retries++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** {@link #issue} for a call with no return value. Returns true when the claim was lost. */
|
||||
private static boolean issueVoid(Runnable call) {
|
||||
return issue(
|
||||
() -> {
|
||||
call.run();
|
||||
return Boolean.TRUE;
|
||||
})
|
||||
.lostClaim;
|
||||
}
|
||||
|
||||
/** Sleep before re-issuing a retryable request. Doubles up to {@link #RETRY_BACKOFF_MAX_MS}. */
|
||||
private static void backoff(int attempt) {
|
||||
long delay = RETRY_BACKOFF_BASE_MS << Math.min(attempt, 8);
|
||||
sleep(Math.min(delay, RETRY_BACKOFF_MAX_MS));
|
||||
}
|
||||
|
||||
private static void sleep(long millis) {
|
||||
try {
|
||||
Thread.sleep(millis);
|
||||
} catch (InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("Interrupted while waiting on the LSM checkpoint", e);
|
||||
}
|
||||
}
|
||||
|
||||
private String route(String operation) {
|
||||
return "/v1/table/" + tableIdentifier + "/" + operation + "/";
|
||||
}
|
||||
|
||||
/** What one LSM request produced: its value, or word that the owning node holds no claim. */
|
||||
private static final class Attempt<T> {
|
||||
private final T value;
|
||||
private final boolean lostClaim;
|
||||
|
||||
private Attempt(T value, boolean lostClaim) {
|
||||
this.value = value;
|
||||
this.lostClaim = lostClaim;
|
||||
}
|
||||
}
|
||||
|
||||
@FunctionalInterface
|
||||
private interface Call<T> {
|
||||
T run();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Live per-bucket LSM state, as returned by {@link LanceDbTableLsm#getLsmStats()}.
|
||||
*
|
||||
* <p>Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are the caller's to
|
||||
* compute. There is no "LSM is off" shape — that case is an empty {@link java.util.Optional},
|
||||
* because a stats object of zeros would read as measurements.
|
||||
*/
|
||||
public class LsmStats {
|
||||
private static final String CONTEXT = "lsm stats";
|
||||
|
||||
private final List<BucketStats> buckets;
|
||||
|
||||
LsmStats(List<BucketStats> buckets) {
|
||||
this.buckets = Collections.unmodifiableList(buckets);
|
||||
}
|
||||
|
||||
/** One entry per bucket. */
|
||||
public List<BucketStats> buckets() {
|
||||
return buckets;
|
||||
}
|
||||
|
||||
static LsmStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
List<BucketStats> buckets = new ArrayList<BucketStats>();
|
||||
for (JsonNode bucket : JsonFields.requiredArray(node, "buckets", CONTEXT)) {
|
||||
buckets.add(BucketStats.fromJson(bucket));
|
||||
}
|
||||
return new LsmStats(buckets);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "LsmStats{buckets=" + buckets + "}";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,260 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Specification selecting Lance's MemWAL LSM-style write path for {@code mergeInsert}.
|
||||
*
|
||||
* <p>Construct via {@link #bucket}, {@link #identity}, or {@link #unsharded}, then optionally chain
|
||||
* {@link #withMaintainedIndexes} and {@link #withWriterConfigDefaults}. Install it with {@link
|
||||
* LanceDbTableLsm#setLsmWriteSpec} and remove it with {@link LanceDbTableLsm#unsetLsmWriteSpec}.
|
||||
*
|
||||
* <p>This is deliberately not {@code org.lance.memwal.InitializeMemWalParams}. That type is Lance's
|
||||
* own, and its maintained-index default is the opposite of this one: it defaults to maintaining
|
||||
* <em>nothing</em>, while a fresh spec here maintains <em>every</em> index. It also cannot express
|
||||
* the null that asks the server to resolve the set.
|
||||
*/
|
||||
public class LsmWriteSpec {
|
||||
|
||||
/** How writes are routed to MemWAL shards. */
|
||||
public enum Sharding {
|
||||
/** Hash-bucket writes by a scalar column. */
|
||||
BUCKET("bucket"),
|
||||
/** Shard by the raw value of a scalar column. */
|
||||
IDENTITY("identity"),
|
||||
/** Route every write to a single shard. */
|
||||
UNSHARDED("unsharded");
|
||||
|
||||
private final String wireName;
|
||||
|
||||
Sharding(String wireName) {
|
||||
this.wireName = wireName;
|
||||
}
|
||||
|
||||
String wireName() {
|
||||
return wireName;
|
||||
}
|
||||
|
||||
static Sharding fromWireName(String name) {
|
||||
for (Sharding s : values()) {
|
||||
if (s.wireName.equals(name)) {
|
||||
return s;
|
||||
}
|
||||
}
|
||||
throw new IllegalArgumentException("Unknown sharding mode: " + name);
|
||||
}
|
||||
}
|
||||
|
||||
private final Sharding sharding;
|
||||
private final String column;
|
||||
private final Integer numBuckets;
|
||||
private final List<String> maintainedIndexes;
|
||||
private final Map<String, String> writerConfigDefaults;
|
||||
|
||||
private LsmWriteSpec(
|
||||
Sharding sharding,
|
||||
String column,
|
||||
Integer numBuckets,
|
||||
List<String> maintainedIndexes,
|
||||
Map<String, String> writerConfigDefaults) {
|
||||
this.sharding = sharding;
|
||||
this.column = column;
|
||||
this.numBuckets = numBuckets;
|
||||
this.maintainedIndexes = maintainedIndexes;
|
||||
this.writerConfigDefaults = writerConfigDefaults;
|
||||
}
|
||||
|
||||
/**
|
||||
* Hash-bucket sharding by a scalar column, maintaining every index on the table.
|
||||
*
|
||||
* <p>Iceberg-compatible Murmur3-x86-32 (seed 0) is used, so each row's {@code bucket(column,
|
||||
* numBuckets)} value is stable across processes.
|
||||
*
|
||||
* @param column A non-nested column with a supported scalar type.
|
||||
* @param numBuckets The number of buckets, in {@code [1, 1024]}.
|
||||
*/
|
||||
public static LsmWriteSpec bucket(String column, int numBuckets) {
|
||||
if (column == null || column.trim().isEmpty()) {
|
||||
throw new IllegalArgumentException("Column cannot be null or empty");
|
||||
}
|
||||
return new LsmWriteSpec(
|
||||
Sharding.BUCKET, column, numBuckets, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
/**
|
||||
* Identity sharding — shard by the raw value of {@code column} — maintaining every index on the
|
||||
* table.
|
||||
*
|
||||
* <p>{@code column} must be a deterministic function of the unenforced primary key: every row
|
||||
* with a given primary key must always produce the same {@code column} value, or upserts of that
|
||||
* key can land in different shards and a stale version can win.
|
||||
*/
|
||||
public static LsmWriteSpec identity(String column) {
|
||||
if (column == null || column.trim().isEmpty()) {
|
||||
throw new IllegalArgumentException("Column cannot be null or empty");
|
||||
}
|
||||
return new LsmWriteSpec(Sharding.IDENTITY, column, null, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
/** No sharding — every write goes to a single MemWAL shard — maintaining every index. */
|
||||
public static LsmWriteSpec unsharded() {
|
||||
return new LsmWriteSpec(Sharding.UNSHARDED, null, null, null, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
/**
|
||||
* Set the indexes the MemWAL keeps up to date as rows are appended.
|
||||
*
|
||||
* <p>Pass {@code null} — the default for a fresh spec — to maintain every index the MemWAL can,
|
||||
* resolved when the spec is installed. That is a snapshot: indexes created later are not
|
||||
* maintained until the spec is unset and set again. Pass an empty list to maintain none.
|
||||
*
|
||||
* <p>Note that {@code null} and the empty list mean opposite things here.
|
||||
*/
|
||||
public LsmWriteSpec withMaintainedIndexes(List<String> maintainedIndexes) {
|
||||
return new LsmWriteSpec(
|
||||
sharding,
|
||||
column,
|
||||
numBuckets,
|
||||
maintainedIndexes == null ? null : new ArrayList<String>(maintainedIndexes),
|
||||
writerConfigDefaults);
|
||||
}
|
||||
|
||||
/**
|
||||
* Set default {@code ShardWriter} configuration recorded in the MemWAL index.
|
||||
*
|
||||
* <p>A sparse override map — only the keys you set are recorded. Recognized keys include {@code
|
||||
* durable_write}, {@code max_wal_buffer_size}, {@code max_memtable_size}, {@code
|
||||
* max_memtable_rows}, {@code max_memtable_batches}, {@code manifest_scan_batch_size}, {@code
|
||||
* max_unflushed_memtable_bytes}, and {@code enable_memtable}. Duration knobs carry an {@code _ms}
|
||||
* suffix, such as {@code max_wal_flush_interval_ms}.
|
||||
*/
|
||||
public LsmWriteSpec withWriterConfigDefaults(Map<String, String> writerConfigDefaults) {
|
||||
if (writerConfigDefaults == null) {
|
||||
throw new IllegalArgumentException("writerConfigDefaults cannot be null");
|
||||
}
|
||||
return new LsmWriteSpec(
|
||||
sharding,
|
||||
column,
|
||||
numBuckets,
|
||||
maintainedIndexes,
|
||||
new HashMap<String, String>(writerConfigDefaults));
|
||||
}
|
||||
|
||||
/** How writes are routed to shards. */
|
||||
public Sharding sharding() {
|
||||
return sharding;
|
||||
}
|
||||
|
||||
/** The sharding column for {@link Sharding#BUCKET} and {@link Sharding#IDENTITY}, else null. */
|
||||
public String column() {
|
||||
return column;
|
||||
}
|
||||
|
||||
/** The bucket count for {@link Sharding#BUCKET}, else null. */
|
||||
public Integer numBuckets() {
|
||||
return numBuckets;
|
||||
}
|
||||
|
||||
/**
|
||||
* The indexes the MemWAL maintains, or null to have the server resolve every maintainable index
|
||||
* on install. An empty list means none.
|
||||
*/
|
||||
public List<String> maintainedIndexes() {
|
||||
return maintainedIndexes == null ? null : Collections.unmodifiableList(maintainedIndexes);
|
||||
}
|
||||
|
||||
/** Default {@code ShardWriter} configuration recorded in the MemWAL index. */
|
||||
public Map<String, String> writerConfigDefaults() {
|
||||
return Collections.unmodifiableMap(writerConfigDefaults);
|
||||
}
|
||||
|
||||
/** Render this spec as the {@code set_lsm_write_spec} request body. */
|
||||
Map<String, Object> toRequestBody() {
|
||||
Map<String, Object> shardingBody = new LinkedHashMap<String, Object>();
|
||||
shardingBody.put("mode", sharding.wireName());
|
||||
if (column != null) {
|
||||
shardingBody.put("column", column);
|
||||
}
|
||||
if (numBuckets != null) {
|
||||
shardingBody.put("num_buckets", numBuckets);
|
||||
}
|
||||
|
||||
Map<String, Object> body = new LinkedHashMap<String, Object>();
|
||||
body.put("sharding", shardingBody);
|
||||
// Null is meaningful: it asks the server to resolve every maintainable index.
|
||||
body.put("maintained_indexes", maintainedIndexes);
|
||||
body.put("writer_config_defaults", writerConfigDefaults);
|
||||
return body;
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild a spec from a {@code get_lsm_write_spec} response body.
|
||||
*
|
||||
* <p>The server always reports a concrete maintained-index list, so a null selection never
|
||||
* round-trips.
|
||||
*/
|
||||
static LsmWriteSpec fromJson(JsonNode node) {
|
||||
JsonNode shardingNode = node.get("sharding");
|
||||
if (shardingNode == null || shardingNode.get("mode") == null) {
|
||||
throw new IllegalStateException("get_lsm_write_spec response has no sharding mode");
|
||||
}
|
||||
Sharding sharding = Sharding.fromWireName(shardingNode.get("mode").asText());
|
||||
|
||||
String column = shardingNode.hasNonNull("column") ? shardingNode.get("column").asText() : null;
|
||||
Integer numBuckets =
|
||||
shardingNode.hasNonNull("num_buckets") ? shardingNode.get("num_buckets").asInt() : null;
|
||||
|
||||
List<String> maintainedIndexes = new ArrayList<String>();
|
||||
JsonNode indexesNode = node.get("maintained_indexes");
|
||||
if (indexesNode != null && indexesNode.isArray()) {
|
||||
for (JsonNode index : indexesNode) {
|
||||
maintainedIndexes.add(index.asText());
|
||||
}
|
||||
}
|
||||
|
||||
Map<String, String> defaults = new HashMap<String, String>();
|
||||
JsonNode defaultsNode = node.get("writer_config_defaults");
|
||||
if (defaultsNode != null && defaultsNode.isObject()) {
|
||||
defaultsNode
|
||||
.fieldNames()
|
||||
.forEachRemaining(name -> defaults.put(name, defaultsNode.get(name).asText()));
|
||||
}
|
||||
|
||||
return new LsmWriteSpec(sharding, column, numBuckets, maintainedIndexes, defaults);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "LsmWriteSpec{sharding="
|
||||
+ sharding
|
||||
+ ", column="
|
||||
+ column
|
||||
+ ", numBuckets="
|
||||
+ numBuckets
|
||||
+ ", maintainedIndexes="
|
||||
+ maintainedIndexes
|
||||
+ ", writerConfigDefaults="
|
||||
+ writerConfigDefaults
|
||||
+ "}";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
||||
/** One in-memory memtable. */
|
||||
public class MemtableStats {
|
||||
private static final String CONTEXT = "memtable stats";
|
||||
|
||||
private final long generation;
|
||||
private final long rows;
|
||||
private final long bytes;
|
||||
private final long batches;
|
||||
private final List<String> indexes;
|
||||
|
||||
MemtableStats(long generation, long rows, long bytes, long batches, List<String> indexes) {
|
||||
this.generation = generation;
|
||||
this.rows = rows;
|
||||
this.bytes = bytes;
|
||||
this.batches = batches;
|
||||
this.indexes = Collections.unmodifiableList(indexes);
|
||||
}
|
||||
|
||||
/** The generation this memtable will become once sealed. */
|
||||
public long generation() {
|
||||
return generation;
|
||||
}
|
||||
|
||||
/** Rows currently buffered. */
|
||||
public long rows() {
|
||||
return rows;
|
||||
}
|
||||
|
||||
/** Estimated in-memory size. */
|
||||
public long bytes() {
|
||||
return bytes;
|
||||
}
|
||||
|
||||
/** Record batches currently buffered. */
|
||||
public long batches() {
|
||||
return batches;
|
||||
}
|
||||
|
||||
/**
|
||||
* Names of the indexes this memtable carries. An absent name is the whole answer to "why is my
|
||||
* fresh-tier search on that column brute-force".
|
||||
*/
|
||||
public List<String> indexes() {
|
||||
return indexes;
|
||||
}
|
||||
|
||||
static MemtableStats fromJson(JsonNode node) {
|
||||
JsonFields.requiredObject(node, CONTEXT);
|
||||
List<String> indexes = new ArrayList<String>();
|
||||
for (JsonNode index : JsonFields.requiredArray(node, "indexes", CONTEXT)) {
|
||||
if (!index.isTextual()) {
|
||||
throw new IllegalStateException(CONTEXT + " has a non-string index name: " + index);
|
||||
}
|
||||
indexes.add(index.asText());
|
||||
}
|
||||
return new MemtableStats(
|
||||
JsonFields.requiredLong(node, "generation", CONTEXT),
|
||||
JsonFields.requiredLong(node, "rows", CONTEXT),
|
||||
JsonFields.requiredLong(node, "bytes", CONTEXT),
|
||||
JsonFields.requiredLong(node, "batches", CONTEXT),
|
||||
indexes);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "MemtableStats{generation="
|
||||
+ generation
|
||||
+ ", rows="
|
||||
+ rows
|
||||
+ ", bytes="
|
||||
+ bytes
|
||||
+ ", batches="
|
||||
+ batches
|
||||
+ ", indexes="
|
||||
+ indexes
|
||||
+ "}";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,570 @@
|
||||
/*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package com.lancedb;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.UncheckedIOException;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayDeque;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.Deque;
|
||||
import java.util.HashMap;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* Unit tests for the MemWAL LSM routes, run against a scripted local HTTP server.
|
||||
*
|
||||
* <p>The wire assertions mirror the Rust mocked-endpoint tests in {@code
|
||||
* rust/lancedb/src/remote/table.rs}, which are the contract these routes have to match.
|
||||
*/
|
||||
public class LanceDbTableLsmTest {
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
|
||||
private HttpServer server;
|
||||
private LanceDbRestClient client;
|
||||
private LanceDbTableLsm lsm;
|
||||
|
||||
private final List<String> requestPaths = Collections.synchronizedList(new ArrayList<String>());
|
||||
private final List<String> requestBodies = Collections.synchronizedList(new ArrayList<String>());
|
||||
private final Map<String, Deque<Reply>> replies = new ConcurrentHashMap<String, Deque<Reply>>();
|
||||
|
||||
@BeforeEach
|
||||
public void setUp() throws IOException {
|
||||
start();
|
||||
}
|
||||
|
||||
/** Tear down and restart the scripted server, for a test that scripts several exchanges. */
|
||||
private void setUpFresh() {
|
||||
try {
|
||||
client.close();
|
||||
server.stop(0);
|
||||
requestPaths.clear();
|
||||
requestBodies.clear();
|
||||
replies.clear();
|
||||
start();
|
||||
} catch (IOException e) {
|
||||
throw new UncheckedIOException(e);
|
||||
}
|
||||
}
|
||||
|
||||
private void start() throws IOException {
|
||||
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
|
||||
server.createContext(
|
||||
"/",
|
||||
exchange -> {
|
||||
String path = exchange.getRequestURI().getPath();
|
||||
requestPaths.add(path);
|
||||
requestBodies.add(readAll(exchange.getRequestBody()));
|
||||
|
||||
Reply reply = nextReply(path);
|
||||
byte[] out = reply.body.getBytes(StandardCharsets.UTF_8);
|
||||
exchange.sendResponseHeaders(reply.status, out.length == 0 ? -1 : out.length);
|
||||
if (out.length > 0) {
|
||||
exchange.getResponseBody().write(out);
|
||||
}
|
||||
exchange.close();
|
||||
});
|
||||
server.start();
|
||||
|
||||
client =
|
||||
LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.apiKey("test-key")
|
||||
.database("test-db")
|
||||
.endpoint("http://127.0.0.1:" + server.getAddress().getPort())
|
||||
.buildRestClient();
|
||||
lsm = new LanceDbTableLsm(client, "my_table");
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
public void tearDown() throws IOException {
|
||||
client.close();
|
||||
server.stop(0);
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// set / unset / get spec
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecUnsharded() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded());
|
||||
|
||||
assertEquals("/v1/table/my_table/set_lsm_write_spec/", requestPaths.get(0));
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("unsharded", body.get("sharding").get("mode").asText());
|
||||
assertFalse(body.get("sharding").has("column"));
|
||||
assertFalse(body.get("sharding").has("num_buckets"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecBucket() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(
|
||||
LsmWriteSpec.bucket("id", 16).withMaintainedIndexes(Arrays.asList("id_idx")));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("bucket", body.get("sharding").get("mode").asText());
|
||||
assertEquals("id", body.get("sharding").get("column").asText());
|
||||
assertEquals(16, body.get("sharding").get("num_buckets").asInt());
|
||||
assertEquals(1, body.get("maintained_indexes").size());
|
||||
assertEquals("id_idx", body.get("maintained_indexes").get(0).asText());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecIdentity() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.identity("tenant"));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("identity", body.get("sharding").get("mode").asText());
|
||||
assertEquals("tenant", body.get("sharding").get("column").asText());
|
||||
assertFalse(body.get("sharding").has("num_buckets"));
|
||||
}
|
||||
|
||||
/**
|
||||
* The tri-state that motivated a LanceDB-owned spec type: a null selection asks the server to
|
||||
* resolve every maintainable index, while an empty list asks for none. They must not collapse.
|
||||
*/
|
||||
@Test
|
||||
public void testMaintainedIndexesNullAndEmptyAreDistinctOnTheWire() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded());
|
||||
JsonNode fresh = MAPPER.readTree(requestBodies.get(0));
|
||||
assertTrue(fresh.has("maintained_indexes"), "the key must be present");
|
||||
assertTrue(fresh.get("maintained_indexes").isNull(), "a fresh spec sends null, not []");
|
||||
|
||||
lsm.setLsmWriteSpec(
|
||||
LsmWriteSpec.unsharded().withMaintainedIndexes(Collections.<String>emptyList()));
|
||||
JsonNode none = MAPPER.readTree(requestBodies.get(1));
|
||||
assertTrue(none.get("maintained_indexes").isArray());
|
||||
assertEquals(0, none.get("maintained_indexes").size());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLsmWriteSpecWriterConfigDefaults() throws Exception {
|
||||
enqueue("set_lsm_write_spec", 200, "");
|
||||
|
||||
Map<String, String> defaults = new HashMap<String, String>();
|
||||
defaults.put("max_memtable_rows", "50000");
|
||||
lsm.setLsmWriteSpec(LsmWriteSpec.unsharded().withWriterConfigDefaults(defaults));
|
||||
|
||||
JsonNode body = MAPPER.readTree(requestBodies.get(0));
|
||||
assertEquals("50000", body.get("writer_config_defaults").get("max_memtable_rows").asText());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testUnsetLsmWriteSpec() {
|
||||
enqueue("unset_lsm_write_spec", 200, "");
|
||||
|
||||
lsm.unsetLsmWriteSpec();
|
||||
|
||||
assertEquals("/v1/table/my_table/unset_lsm_write_spec/", requestPaths.get(0));
|
||||
assertEquals("", requestBodies.get(0));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpec() {
|
||||
enqueue(
|
||||
"get_lsm_write_spec",
|
||||
200,
|
||||
"{\"lsm_write_spec\":{\"sharding\":{\"mode\":\"bucket\",\"column\":\"id\","
|
||||
+ "\"num_buckets\":16},\"maintained_indexes\":[\"id_idx\"],"
|
||||
+ "\"writer_config_defaults\":{\"durable_write\":\"true\"}}}");
|
||||
|
||||
Optional<LsmWriteSpec> spec = lsm.getLsmWriteSpec();
|
||||
|
||||
assertTrue(spec.isPresent());
|
||||
assertEquals(LsmWriteSpec.Sharding.BUCKET, spec.get().sharding());
|
||||
assertEquals("id", spec.get().column());
|
||||
assertEquals(Integer.valueOf(16), spec.get().numBuckets());
|
||||
assertEquals(Arrays.asList("id_idx"), spec.get().maintainedIndexes());
|
||||
assertEquals("true", spec.get().writerConfigDefaults().get("durable_write"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmWriteSpecAbsent() {
|
||||
enqueue("get_lsm_write_spec", 200, "{\"lsm_write_spec\":null}");
|
||||
|
||||
assertFalse(lsm.getLsmWriteSpec().isPresent());
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// stats
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testGetLsmStats() throws Exception {
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
|
||||
Optional<LsmStats> got = lsm.getLsmStats(true);
|
||||
|
||||
assertEquals("/v1/table/my_table/get_lsm_stats/", requestPaths.get(0));
|
||||
assertTrue(MAPPER.readTree(requestBodies.get(0)).get("include_generation_rows").asBoolean());
|
||||
assertTrue(got.isPresent());
|
||||
BucketStats decoded = got.get().buckets().get(0);
|
||||
assertEquals("shard-0", decoded.shardId());
|
||||
assertEquals("Active", decoded.status());
|
||||
assertEquals(1, decoded.writerEpoch());
|
||||
assertEquals(2, decoded.manifestVersion());
|
||||
assertEquals(9, decoded.currentGeneration());
|
||||
assertFalse(decoded.compacting());
|
||||
assertEquals(Arrays.asList(7L, 8L), generationNumbers(decoded));
|
||||
assertEquals(1024, decoded.generations().get(0).bytes());
|
||||
assertFalse(decoded.generations().get(0).rows().isPresent(), "rows absent unless requested");
|
||||
assertFalse(decoded.memtables().isPresent(), "absent memtables stay absent");
|
||||
}
|
||||
|
||||
/** The optional fields decode when the server does send them. */
|
||||
@Test
|
||||
public void testGetLsmStatsDecodesOptionalFields() {
|
||||
enqueue(
|
||||
"get_lsm_stats",
|
||||
200,
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":3,\"wal_entry_position_last_seen\":11,"
|
||||
+ "\"generations\":[{\"generation\":7,\"bytes\":1024,\"rows\":42}],"
|
||||
+ "\"compacting\":true,\"memtables\":[{\"generation\":8,\"rows\":5,"
|
||||
+ "\"bytes\":64,\"batches\":2,\"indexes\":[\"id_idx\"]}]}]}}");
|
||||
|
||||
BucketStats decoded = lsm.getLsmStats(true).get().buckets().get(0);
|
||||
|
||||
assertEquals(3, decoded.replayAfterWalEntryPosition());
|
||||
assertEquals(11, decoded.walEntryPositionLastSeen());
|
||||
assertTrue(decoded.compacting());
|
||||
assertEquals(42, decoded.generations().get(0).rows().getAsLong());
|
||||
assertTrue(decoded.memtables().isPresent());
|
||||
MemtableStats memtable = decoded.memtables().get().get(0);
|
||||
assertEquals(8, memtable.generation());
|
||||
assertEquals(5, memtable.rows());
|
||||
assertEquals(64, memtable.bytes());
|
||||
assertEquals(2, memtable.batches());
|
||||
assertEquals(Arrays.asList("id_idx"), memtable.indexes());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmStatsAbsentWhenLsmDisabled() {
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
|
||||
assertFalse(lsm.getLsmStats().isPresent());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetLsmStatsDefaultsToExcludingGenerationRows() throws Exception {
|
||||
enqueue("get_lsm_stats", 200, stats());
|
||||
|
||||
lsm.getLsmStats();
|
||||
|
||||
assertFalse(MAPPER.readTree(requestBodies.get(0)).get("include_generation_rows").asBoolean());
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// flush / compact
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testFlushAndCompactRoutes() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("compact_lsm", 200, "");
|
||||
|
||||
lsm.flushLsm();
|
||||
lsm.compactLsm();
|
||||
|
||||
assertEquals("/v1/table/my_table/flush_lsm/", requestPaths.get(0));
|
||||
assertEquals("/v1/table/my_table/compact_lsm/", requestPaths.get(1));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testHttpErrorCarriesStatus() {
|
||||
enqueue("flush_lsm", 404, "no such table");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.flushLsm());
|
||||
assertEquals(404, e.statusCode());
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// checkpoint
|
||||
// ===========================================================================
|
||||
|
||||
@Test
|
||||
public void testCheckpointReturnsWhenLsmDisabled() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"), "nothing to compact when the LSM path is off");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointReturnsWhenNoGenerationsOutstanding() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
// A bucket with no L0 generations yields no target, so the drain never starts.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointConvergesOnceTargetGenerationsAreGone() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
// Watermark read: shard-0 holds generations 7 and 8, so target = 8.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
// First drain poll: both still outstanding, nothing compacting -> dispatch a pass.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 7L, 8L)));
|
||||
// Second drain poll: drained past the target -> done.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 9L)));
|
||||
enqueue("compact_lsm", 200, "");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
|
||||
assertEquals(1, countCalls("compact_lsm"), "one pass dispatched");
|
||||
assertEquals(3, countCalls("get_lsm_stats"), "watermark read plus two drain polls");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointDoesNotPileOnWhileEveryTargetBucketIsCompacting() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", true, 4L)));
|
||||
// Still compacting on the first poll, so no pass is dispatched; then it drains.
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", true, 4L)));
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false, 5L)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
|
||||
assertEquals(0, countCalls("compact_lsm"), "a latched bucket is left alone");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointRetriesFromFlushAfterLostClaim() {
|
||||
// 421 on the watermark read: the node lost its claim, so the whole thing restarts
|
||||
// from flush rather than retrying the read in place.
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 421, "no claim");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
|
||||
assertEquals(2, countCalls("flush_lsm"), "re-issued from flush");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointRetriesRetryableStatusInPlace() {
|
||||
enqueue("flush_lsm", 429, "latch held");
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, stats(bucket("shard-0", false)));
|
||||
|
||||
lsm.checkpointLsm();
|
||||
|
||||
assertEquals(2, countCalls("flush_lsm"), "429 retried in place, not re-issued");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointPropagatesTerminalStatus() {
|
||||
enqueue("flush_lsm", 400, "bad request");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.checkpointLsm());
|
||||
assertEquals(400, e.statusCode());
|
||||
assertEquals(1, countCalls("flush_lsm"), "a terminal status is not retried");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckpointGivesUpAfterRepeatedLostClaims() {
|
||||
enqueue("flush_lsm", 421, "no claim");
|
||||
|
||||
IllegalStateException e = assertThrows(IllegalStateException.class, () -> lsm.checkpointLsm());
|
||||
assertTrue(e.getMessage().contains("kept losing its claim"), e.getMessage());
|
||||
assertEquals(4, countCalls("flush_lsm"), "the initial attempt plus MAX_REISSUES");
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// strict decoding
|
||||
// ===========================================================================
|
||||
|
||||
/**
|
||||
* A stats payload that does not decode must fail closed. Every one of these bodies used to be
|
||||
* read as "no buckets", which is indistinguishable from a drained table, so {@code checkpointLsm}
|
||||
* reported convergence for a checkpoint that never ran.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckpointRejectsMalformedStats() {
|
||||
Map<String, String> malformed = new LinkedHashMap<String, String>();
|
||||
malformed.put("no response body at all", "");
|
||||
malformed.put("stats object with no buckets", "{\"lsm_stats\":{}}");
|
||||
malformed.put("bucket missing its required fields", "{\"lsm_stats\":{\"buckets\":[{}]}}");
|
||||
malformed.put(
|
||||
"bucket missing generations",
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
|
||||
+ "\"compacting\":false}]}}");
|
||||
malformed.put(
|
||||
"generation with a non-numeric generation number",
|
||||
"{\"lsm_stats\":{\"buckets\":[{\"shard_id\":\"shard-0\",\"status\":\"Active\","
|
||||
+ "\"writer_epoch\":1,\"manifest_version\":2,\"current_generation\":9,"
|
||||
+ "\"replay_after_wal_entry_position\":0,\"wal_entry_position_last_seen\":0,"
|
||||
+ "\"generations\":[{\"generation\":\"7\",\"bytes\":1024}],"
|
||||
+ "\"compacting\":false}]}}");
|
||||
|
||||
for (Map.Entry<String, String> each : malformed.entrySet()) {
|
||||
setUpFresh();
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, each.getValue());
|
||||
|
||||
assertThrows(
|
||||
IllegalStateException.class,
|
||||
() -> lsm.checkpointLsm(),
|
||||
each.getKey() + " must not report convergence");
|
||||
}
|
||||
}
|
||||
|
||||
/** The one shape that legitimately means "this table has no LSM write path". */
|
||||
@Test
|
||||
public void testCheckpointTreatsNullStatsAsNotWalBacked() {
|
||||
enqueue("flush_lsm", 200, "");
|
||||
enqueue("get_lsm_stats", 200, "{\"lsm_stats\":null}");
|
||||
|
||||
lsm.checkpointLsm();
|
||||
|
||||
assertEquals(1, countCalls("get_lsm_stats"));
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// retry budget
|
||||
// ===========================================================================
|
||||
|
||||
/**
|
||||
* The transport must not retry on the checkpoint loop's behalf. Apache HttpClient's default
|
||||
* strategy retries exactly 429 and 503 — the two statuses {@code isRetryable} owns — which
|
||||
* doubled every budget here and also retried {@code compact_lsm} in place, where the loop is
|
||||
* built to fall through to a fresh stats poll instead.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckpointRetryBudgetIsNotDoubledByTheTransport() {
|
||||
enqueue("flush_lsm", 429, "latch held");
|
||||
|
||||
LanceDbRestClient.HttpException e =
|
||||
assertThrows(LanceDbRestClient.HttpException.class, () -> lsm.checkpointLsm());
|
||||
|
||||
assertEquals(429, e.statusCode(), "the exhausted budget propagates the last error as itself");
|
||||
assertEquals(9, countCalls("flush_lsm"), "the initial request plus MAX_RETRIES, and no more");
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// harness
|
||||
// ===========================================================================
|
||||
|
||||
private static List<Long> generationNumbers(BucketStats bucket) {
|
||||
List<Long> numbers = new ArrayList<Long>();
|
||||
for (GenerationStats generation : bucket.generations()) {
|
||||
numbers.add(generation.generation());
|
||||
}
|
||||
return numbers;
|
||||
}
|
||||
|
||||
/** Build an {@code lsm_stats} response body from bucket fragments. */
|
||||
private static String stats(String... buckets) {
|
||||
return "{\"lsm_stats\":{\"buckets\":[" + String.join(",", buckets) + "]}}";
|
||||
}
|
||||
|
||||
private static String bucket(String shardId, boolean compacting, Long... generations) {
|
||||
StringBuilder gens = new StringBuilder();
|
||||
for (Long generation : generations) {
|
||||
if (gens.length() > 0) {
|
||||
gens.append(",");
|
||||
}
|
||||
gens.append("{\"generation\":").append(generation).append(",\"bytes\":1024}");
|
||||
}
|
||||
return "{\"shard_id\":\""
|
||||
+ shardId
|
||||
+ "\",\"status\":\"Active\",\"writer_epoch\":1,\"manifest_version\":2,"
|
||||
+ "\"current_generation\":9,\"replay_after_wal_entry_position\":0,"
|
||||
+ "\"wal_entry_position_last_seen\":0,\"generations\":["
|
||||
+ gens
|
||||
+ "],\"compacting\":"
|
||||
+ compacting
|
||||
+ "}";
|
||||
}
|
||||
|
||||
/** Queue a reply for an operation. The last queued reply repeats once the queue drains. */
|
||||
private void enqueue(String operation, int status, String body) {
|
||||
replies.computeIfAbsent(operation, key -> new ArrayDeque<Reply>()).add(new Reply(status, body));
|
||||
}
|
||||
|
||||
private Reply nextReply(String path) {
|
||||
String operation = operationOf(path);
|
||||
Deque<Reply> queued = replies.get(operation);
|
||||
if (queued == null || queued.isEmpty()) {
|
||||
return new Reply(200, "");
|
||||
}
|
||||
return queued.size() > 1 ? queued.poll() : queued.peek();
|
||||
}
|
||||
|
||||
private long countCalls(String operation) {
|
||||
return requestPaths.stream().filter(path -> operationOf(path).equals(operation)).count();
|
||||
}
|
||||
|
||||
/** {@code /v1/table/my_table/flush_lsm/} -> {@code flush_lsm}. */
|
||||
private static String operationOf(String path) {
|
||||
String[] segments = path.split("/");
|
||||
return segments.length == 0 ? "" : segments[segments.length - 1];
|
||||
}
|
||||
|
||||
private static String readAll(InputStream in) throws IOException {
|
||||
ByteArrayOutputStream out = new ByteArrayOutputStream();
|
||||
byte[] buffer = new byte[4096];
|
||||
int read;
|
||||
while ((read = in.read(buffer)) != -1) {
|
||||
out.write(buffer, 0, read);
|
||||
}
|
||||
return new String(out.toByteArray(), StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
private static final class Reply {
|
||||
private final int status;
|
||||
private final String body;
|
||||
|
||||
private Reply(int status, String body) {
|
||||
this.status = status;
|
||||
this.body = body;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3341,6 +3341,59 @@ describe("LSM merge insert", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("LSM convergence and stats", () => {
|
||||
let tmpDir: tmp.DirResult;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = tmp.dirSync({ unsafeCleanup: true });
|
||||
});
|
||||
afterEach(() => tmpDir.removeCallback());
|
||||
|
||||
async function lsmTable(conn: Connection): Promise<Table> {
|
||||
const table = await conn.createEmptyTable(
|
||||
"t",
|
||||
new arrow.Schema([new arrow.Field("id", new arrow.Utf8(), false)]),
|
||||
);
|
||||
await table.setUnenforcedPrimaryKey("id");
|
||||
await table.setLsmWriteSpec({ specType: "unsharded" });
|
||||
return table;
|
||||
}
|
||||
|
||||
// These four route through the server that owns the MemWAL, so a local table
|
||||
// rejects them rather than answering. What is asserted here is that the
|
||||
// bindings reach the core at all; the behavior against a real endpoint is
|
||||
// covered by the mocked endpoint tests in rust/lancedb/src/remote/table.rs.
|
||||
it("rejects flushLsm on a local table", async () => {
|
||||
const conn = await connect(tmpDir.name);
|
||||
const table = await lsmTable(conn);
|
||||
|
||||
await expect(table.flushLsm()).rejects.toThrow(/not supported/i);
|
||||
});
|
||||
|
||||
it("rejects compactLsm on a local table", async () => {
|
||||
const conn = await connect(tmpDir.name);
|
||||
const table = await lsmTable(conn);
|
||||
|
||||
await expect(table.compactLsm()).rejects.toThrow(/not supported/i);
|
||||
});
|
||||
|
||||
it("rejects getLsmStats on a local table", async () => {
|
||||
const conn = await connect(tmpDir.name);
|
||||
const table = await lsmTable(conn);
|
||||
|
||||
await expect(table.getLsmStats()).rejects.toThrow(/not supported/i);
|
||||
await expect(table.getLsmStats(true)).rejects.toThrow(/not supported/i);
|
||||
});
|
||||
|
||||
it("rejects checkpointLsm on a local table", async () => {
|
||||
const conn = await connect(tmpDir.name);
|
||||
const table = await lsmTable(conn);
|
||||
|
||||
// checkpointLsm seals first, so it surfaces flushLsm's rejection.
|
||||
await expect(table.checkpointLsm()).rejects.toThrow(/not supported/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe("computed columns", () => {
|
||||
let tmpDir: tmp.DirResult;
|
||||
beforeEach(() => {
|
||||
|
||||
@@ -147,6 +147,10 @@ export {
|
||||
FtsToken,
|
||||
TokenizeTableOptions,
|
||||
LsmWriteSpec,
|
||||
LsmStats,
|
||||
BucketStats,
|
||||
GenerationStats,
|
||||
MemtableStats,
|
||||
ColumnAlteration,
|
||||
FieldMetadataUpdate,
|
||||
} from "./table";
|
||||
|
||||
@@ -31,6 +31,7 @@ import {
|
||||
IndexConfig,
|
||||
IndexStatistics,
|
||||
Job,
|
||||
LsmStats,
|
||||
Branches as NativeBranches,
|
||||
OptimizeStats,
|
||||
RefreshColumnResult,
|
||||
@@ -50,6 +51,12 @@ import {
|
||||
import { sanitizeType } from "./sanitize";
|
||||
import { IntoSql, toSQL } from "./util";
|
||||
export { IndexConfig } from "./native";
|
||||
export {
|
||||
BucketStats,
|
||||
GenerationStats,
|
||||
LsmStats,
|
||||
MemtableStats,
|
||||
} from "./native";
|
||||
|
||||
/**
|
||||
* Progress snapshot for a write operation, delivered to the `progress`
|
||||
@@ -706,6 +713,59 @@ export abstract class Table {
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
abstract closeLsmWriters(): Promise<void>;
|
||||
/**
|
||||
* Seal every bucket's active memtable into a new L0 generation.
|
||||
*
|
||||
* Returns once the seal is committed. Sealing an empty memtable is a no-op,
|
||||
* so this is safe to call repeatedly.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
abstract flushLsm(): Promise<void>;
|
||||
/**
|
||||
* Trigger a background L0 → base compaction pass per bucket.
|
||||
*
|
||||
* Returns once the passes are *dispatched*, not once they finish — watch
|
||||
* {@link Table#getLsmStats} for progress, or use
|
||||
* {@link Table#checkpointLsm} to wait for convergence.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
abstract compactLsm(): Promise<void>;
|
||||
/**
|
||||
* Converge this table's LSM write path into its base table.
|
||||
*
|
||||
* Seals once, then triggers compaction and polls until the L0 that existed
|
||||
* at the start is gone. The target set is fixed at the start, so
|
||||
* generations created *during* the checkpoint are ignored — that is what
|
||||
* lets it terminate under write load, and what makes it best-effort: it
|
||||
* converges the fresh tier as of some instant. Idempotent, abandonable at
|
||||
* any point, and safe to run on a cadence.
|
||||
*
|
||||
* There is no liveness bound — the compactor pool is shared across tables,
|
||||
* so a checkpoint queued behind unrelated work looks exactly like one that
|
||||
* is merging. The caller owns the deadline.
|
||||
* @returns {Promise<void>}
|
||||
* @example
|
||||
* ```ts
|
||||
* const before = await table.getLsmStats();
|
||||
* await table.checkpointLsm();
|
||||
* const after = await table.getLsmStats();
|
||||
* ```
|
||||
*/
|
||||
abstract checkpointLsm(): Promise<void>;
|
||||
/**
|
||||
* Read live per-bucket LSM state.
|
||||
*
|
||||
* Answers "how far behind is my fresh tier", "which bucket is hot", and
|
||||
* "why is my fresh-tier vector search brute-force". Mutates no table state.
|
||||
*
|
||||
* Resolves to `undefined` only when the LSM write path is not enabled.
|
||||
* @param {boolean} includeGenerationRows Also count rows per L0 generation.
|
||||
* Off by default because each count opens an uncached Lance dataset.
|
||||
* @returns {Promise<LsmStats | undefined>}
|
||||
*/
|
||||
abstract getLsmStats(
|
||||
includeGenerationRows?: boolean,
|
||||
): Promise<LsmStats | undefined>;
|
||||
/** Retrieve the version of the table */
|
||||
|
||||
abstract version(): Promise<number>;
|
||||
@@ -1266,6 +1326,24 @@ export class LocalTable extends Table {
|
||||
return await this.inner.closeLsmWriters();
|
||||
}
|
||||
|
||||
async flushLsm(): Promise<void> {
|
||||
return await this.inner.flushLsm();
|
||||
}
|
||||
|
||||
async compactLsm(): Promise<void> {
|
||||
return await this.inner.compactLsm();
|
||||
}
|
||||
|
||||
async checkpointLsm(): Promise<void> {
|
||||
return await this.inner.checkpointLsm();
|
||||
}
|
||||
|
||||
async getLsmStats(
|
||||
includeGenerationRows: boolean = false,
|
||||
): Promise<LsmStats | undefined> {
|
||||
return (await this.inner.getLsmStats(includeGenerationRows)) ?? undefined;
|
||||
}
|
||||
|
||||
async version(): Promise<number> {
|
||||
return await this.inner.version();
|
||||
}
|
||||
|
||||
@@ -497,6 +497,34 @@ impl Table {
|
||||
self.inner_ref()?.close_lsm_writers().await.default_error()
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn flush_lsm(&self) -> napi::Result<()> {
|
||||
self.inner_ref()?.flush_lsm().await.default_error()
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn compact_lsm(&self) -> napi::Result<()> {
|
||||
self.inner_ref()?.compact_lsm().await.default_error()
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn checkpoint_lsm(&self) -> napi::Result<()> {
|
||||
self.inner_ref()?.checkpoint_lsm().await.default_error()
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn get_lsm_stats(
|
||||
&self,
|
||||
include_generation_rows: bool,
|
||||
) -> napi::Result<Option<LsmStats>> {
|
||||
let stats = self
|
||||
.inner_ref()?
|
||||
.get_lsm_stats(include_generation_rows)
|
||||
.await
|
||||
.default_error()?;
|
||||
Ok(stats.map(LsmStats::from))
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn version(&self) -> napi::Result<i64> {
|
||||
self.inner_ref()?
|
||||
@@ -889,6 +917,129 @@ impl From<lancedb::table::LsmWriteSpec> for LsmWriteSpec {
|
||||
}
|
||||
}
|
||||
|
||||
/// One flushed L0 generation.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct GenerationStats {
|
||||
/// The generation number. Increases as memtables are sealed into L0.
|
||||
pub generation: i64,
|
||||
/// On-disk size of the generation.
|
||||
pub bytes: i64,
|
||||
/// Present only when `includeGenerationRows` was requested. Off by default
|
||||
/// because each count opens an uncached Lance dataset.
|
||||
pub rows: Option<i64>,
|
||||
}
|
||||
|
||||
impl From<lancedb::table::GenerationStats> for GenerationStats {
|
||||
fn from(g: lancedb::table::GenerationStats) -> Self {
|
||||
Self {
|
||||
generation: g.generation as i64,
|
||||
bytes: g.bytes as i64,
|
||||
rows: g.rows.map(|r| r as i64),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One in-memory memtable.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct MemtableStats {
|
||||
/// The generation this memtable will become once sealed.
|
||||
pub generation: i64,
|
||||
/// Rows currently buffered.
|
||||
pub rows: i64,
|
||||
/// Estimated in-memory size.
|
||||
pub bytes: i64,
|
||||
/// Record batches currently buffered.
|
||||
pub batches: i64,
|
||||
/// Names of the indexes this memtable carries. An absent name is the whole
|
||||
/// answer to "why is my fresh-tier search on that column brute-force".
|
||||
pub indexes: Vec<String>,
|
||||
}
|
||||
|
||||
impl From<lancedb::table::MemtableStats> for MemtableStats {
|
||||
fn from(m: lancedb::table::MemtableStats) -> Self {
|
||||
Self {
|
||||
generation: m.generation as i64,
|
||||
rows: m.rows as i64,
|
||||
bytes: m.bytes as i64,
|
||||
batches: m.batches as i64,
|
||||
indexes: m.indexes,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Live state of one bucket. A table is N buckets on one node; flattening to a
|
||||
/// single number hides the one hot bucket that is usually why someone opened
|
||||
/// this endpoint.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct BucketStats {
|
||||
/// The shard this bucket writes.
|
||||
pub shard_id: String,
|
||||
/// `"Active"` or `"Sealed"` (drop-table 2PC in flight).
|
||||
pub status: String,
|
||||
/// Epoch of the writer that currently owns the shard.
|
||||
pub writer_epoch: i64,
|
||||
/// Version of the shard manifest these numbers were read from.
|
||||
pub manifest_version: i64,
|
||||
/// The generation the active memtable will become.
|
||||
pub current_generation: i64,
|
||||
/// WAL position replay resumes from.
|
||||
pub replay_after_wal_entry_position: i64,
|
||||
/// Highest WAL position the writer has seen. The difference against
|
||||
/// `replayAfterWalEntryPosition` is the WAL lag.
|
||||
pub wal_entry_position_last_seen: i64,
|
||||
/// Flushed L0 generations not yet merged into the base table.
|
||||
pub generations: Vec<GenerationStats>,
|
||||
/// Whether a pass owns this bucket's compaction latch right now. Says *a*
|
||||
/// driver is running, not *whose*, and the latch is held from dispatch —
|
||||
/// including while the pass queues for a pod-wide compactor permit. Read it
|
||||
/// as "do not pile on", never as "mine is progressing".
|
||||
pub compacting: bool,
|
||||
/// Oldest first, active last. Absent for a `"Sealed"` bucket, whose
|
||||
/// in-memory state is torn down.
|
||||
pub memtables: Option<Vec<MemtableStats>>,
|
||||
}
|
||||
|
||||
impl From<lancedb::table::BucketStats> for BucketStats {
|
||||
fn from(b: lancedb::table::BucketStats) -> Self {
|
||||
Self {
|
||||
shard_id: b.shard_id,
|
||||
status: b.status,
|
||||
writer_epoch: b.writer_epoch as i64,
|
||||
manifest_version: b.manifest_version as i64,
|
||||
current_generation: b.current_generation as i64,
|
||||
replay_after_wal_entry_position: b.replay_after_wal_entry_position as i64,
|
||||
wal_entry_position_last_seen: b.wal_entry_position_last_seen as i64,
|
||||
generations: b.generations.into_iter().map(Into::into).collect(),
|
||||
compacting: b.compacting,
|
||||
memtables: b
|
||||
.memtables
|
||||
.map(|ms| ms.into_iter().map(Into::into).collect()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Live per-bucket LSM state, as returned by `Table#getLsmStats`.
|
||||
///
|
||||
/// Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are
|
||||
/// the caller's to compute.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct LsmStats {
|
||||
/// One entry per bucket backing this table.
|
||||
pub buckets: Vec<BucketStats>,
|
||||
}
|
||||
|
||||
impl From<lancedb::table::LsmStats> for LsmStats {
|
||||
fn from(stats: lancedb::table::LsmStats) -> Self {
|
||||
Self {
|
||||
buckets: stats.buckets.into_iter().map(Into::into).collect(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Statistics about a compaction operation.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
|
||||
@@ -12,6 +12,7 @@ __version__ = importlib.metadata.version("lancedb")
|
||||
|
||||
from ._lancedb import connect as lancedb_connect
|
||||
from ._lancedb import FtsToken
|
||||
from ._lancedb import LsmWriteSpec
|
||||
from ._lancedb import tokenize as _tokenize
|
||||
from .common import URI, sanitize_uri
|
||||
from urllib.parse import urlparse
|
||||
@@ -518,6 +519,7 @@ __all__ = [
|
||||
"Job",
|
||||
"LanceDBConnection",
|
||||
"LanceNamespaceDBConnection",
|
||||
"LsmWriteSpec",
|
||||
"RemoteDBConnection",
|
||||
"Session",
|
||||
"Table",
|
||||
|
||||
@@ -4801,7 +4801,7 @@ class AsyncTable:
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> from lancedb._lancedb import LsmWriteSpec
|
||||
>>> from lancedb import LsmWriteSpec
|
||||
>>> # table.set_unenforced_primary_key("id")
|
||||
>>> # table.set_lsm_write_spec(LsmWriteSpec.bucket("id", 16))
|
||||
"""
|
||||
|
||||
Reference in New Issue
Block a user