mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-19 20:48:34 +00:00
Daniel Rammer
4a04584ce0
Two fixes to the Java MemWAL LSM surface, both reproduced against the
scripted test server before being fixed.
Transport retries. `HttpClients.createDefault()` installs Apache's default
response retry strategy, whose retryable-status list is exactly 429 and 503
— the two statuses `LanceDbTableLsm.isRetryable` owns. Every explicit retry
budget in `checkpointLsm` was therefore doubled on the wire (a 429 held
against flush issued 18 requests, not 9), and `compactLsm` was retried in
place despite the loop being built to fall through to a fresh stats poll.
Automatic retries are now disabled, so the checkpoint loop is the sole owner
of the 421/429/503 transitions.
Stats decoding. `getLsmStats` read the response with Jackson's `path()`,
which yields a missing node that iterates as an empty array. That made
"malformed" indistinguishable from "no buckets", which is indistinguishable
from "drained" — so an empty response body, a `{"lsm_stats": {}}`, or a
bucket missing its required fields all made `checkpointLsm()` report
convergence for a checkpoint that never ran.
Stats now decode into `LsmStats`, `BucketStats`, `GenerationStats` and
`MemtableStats`, mirroring the Rust structs in
`rust/lancedb/src/table/lsm_stats.rs` and the objects the Node binding
already exposes. Decoding is strict and fails closed, matching the serde
contract on the Rust side: absent or null `lsm_stats` means the LSM write
path is off, and anything else present must decode into the full
bucket/generation shape. `newestGeneration` and `outstandingGenerations`
move onto `BucketStats`, matching Rust's `impl BucketStats`.
This changes `getLsmStats` from `Optional<JsonNode>` to `Optional<LsmStats>`,
which also brings Java to the typed surface Node already had rather than
handing back a raw JSON blob.
Testing: 33 passing in lancedb-core, up from 29. The new tests pin the wire
request count against the retry budget and reject five malformed stats
payloads. `testCheckpointRetriesRetryableStatusInPlace` previously passed on
a transport-absorbed 429 and now exercises the real retry path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
87 lines
2.3 KiB
Markdown
87 lines
2.3 KiB
Markdown
# LanceDB Java Enterprise Client
|
|
|
|
## Configuration and Initialization
|
|
|
|
### LanceDB Cloud
|
|
|
|
For LanceDB Cloud, use the simplified builder API:
|
|
|
|
```java
|
|
import com.lancedb.LanceDbNamespaceClientBuilder;
|
|
import org.lance.namespace.LanceNamespace;
|
|
|
|
// If your DB url is db://example-db, then your database here is example-db
|
|
LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
|
.apiKey("your_lancedb_cloud_api_key")
|
|
.database("your_database_name")
|
|
.build();
|
|
```
|
|
|
|
### LanceDB Enterprise
|
|
|
|
For Enterprise deployments, use your custom endpoint:
|
|
|
|
```java
|
|
LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
|
.apiKey("your_lancedb_enterprise_api_key")
|
|
.database("your_database_name")
|
|
.endpoint("<your_enterprise_endpoint>")
|
|
.build();
|
|
```
|
|
|
|
## MemWAL LSM write path
|
|
|
|
Most table operations reach LanceDB through the `LanceNamespace` above, which is
|
|
generated from the Lance Namespace specification. The MemWAL LSM routes are not part
|
|
of that specification, so they are issued through a separate client:
|
|
|
|
```java
|
|
import com.lancedb.LanceDbRestClient;
|
|
import com.lancedb.LanceDbTableLsm;
|
|
import com.lancedb.LsmWriteSpec;
|
|
|
|
LanceDbRestClient client = LanceDbNamespaceClientBuilder.newBuilder()
|
|
.apiKey("your_lancedb_cloud_api_key")
|
|
.database("your_database_name")
|
|
.buildRestClient();
|
|
|
|
LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");
|
|
|
|
// Route future merge_insert upserts through the MemWAL, hash-bucketed by `id`.
|
|
lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));
|
|
|
|
// ... merge_insert traffic ...
|
|
|
|
// Converge the fresh tier into the base table.
|
|
lsm.checkpointLsm();
|
|
|
|
// Inspect live per-bucket state.
|
|
lsm.getLsmStats().ifPresent(stats -> stats.buckets().forEach(bucket ->
|
|
System.out.println(bucket.shardId() + ": " + bucket.generations().size() + " L0 generations")));
|
|
|
|
client.close();
|
|
```
|
|
|
|
`maintainedIndexes` is tri-state, and the null default is the opposite of what a Java
|
|
reader usually expects:
|
|
|
|
| Value | Meaning |
|
|
| --- | --- |
|
|
| unset (null) | Maintain **every** index the MemWAL can, resolved on install |
|
|
| `Collections.emptyList()` | Maintain **none** |
|
|
| `Arrays.asList("id_idx")` | Maintain exactly those |
|
|
|
|
## Development
|
|
|
|
Build:
|
|
|
|
```shell
|
|
./mvnw install -pl lancedb-core -am
|
|
```
|
|
|
|
Run tests:
|
|
|
|
```shell
|
|
./mvnw test -pl lancedb-core
|
|
```
|