mirror of https://github.com/lancedb/lancedb.git synced 2026-08-18 12:08:35 +00:00

Files

T

Daniel Rammer a651b67c76 feat: bring the MemWAL LSM surface to parity across the SDKs

Four of the eight LSM methods are remote-only in the core: `impl BaseTable
for NativeTable` implements only set/unset/get_lsm_write_spec and
close_lsm_writers, while flush_lsm, compact_lsm and get_lsm_stats fall
through to trait defaults returning NotSupported. That is why Node had
bound the four that work locally and stopped, and why the remaining four
had no binding-level coverage anywhere.

Node: add napi bindings for flush_lsm, compact_lsm, checkpoint_lsm and
get_lsm_stats, with typed LsmStats/BucketStats/GenerationStats/
MemtableStats objects mirroring the existing LsmWriteSpec object in the
same file. Tests assert each binding reaches the core and surfaces
NotSupported locally; behavior against a real endpoint stays covered by
the mocked-endpoint tests in rust/lancedb/src/remote/table.rs.

Python: LsmWriteSpec was importable only from the private lancedb._lancedb
-- it appeared in table.py solely under `if TYPE_CHECKING:`. Export it as
lancedb.LsmWriteSpec, add it to __all__, and list it in the API reference,
which had no mention of it and so rendered it nowhere.

Java: add the LSM routes to lancedb-core. Java reaches LanceDB purely over
REST through the generated namespace client, and these routes are not in
the Lance Namespace spec, so they are issued through a small dedicated
client. LsmWriteSpec is deliberately not org.lance.memwal.
InitializeMemWalParams: that type defaults to maintaining no indexes where
a spec here defaults to maintaining every index, and it cannot express the
null that asks the server to resolve the set. checkpointLsm is ported from
rust/lancedb/src/table/checkpoint.rs with its constants and status
semantics intact -- 429/503 retried in place, 421 restarting from flush.

Note: `mvnw spotless:apply` cannot run on JDK 21 (google-java-format 1.7,
pinned in java/pom.xml, predates JDK 16's compiler API change). This is
pre-existing and reproduces on a pristine main checkout; the Java sources
here were formatted by hand to the checkstyle rules.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-17 14:42:37 -05:00

2.2 KiB

Raw Permalink Blame History

LanceDB Java Enterprise Client

Configuration and Initialization

LanceDB Cloud

For LanceDB Cloud, use the simplified builder API:

import com.lancedb.LanceDbNamespaceClientBuilder;
import org.lance.namespace.LanceNamespace;

// If your DB url is db://example-db, then your database here is example-db
LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
    .apiKey("your_lancedb_cloud_api_key")
    .database("your_database_name")
    .build();

LanceDB Enterprise

For Enterprise deployments, use your custom endpoint:

LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
    .apiKey("your_lancedb_enterprise_api_key")
    .database("your_database_name")
    .endpoint("<your_enterprise_endpoint>")
    .build();

MemWAL LSM write path

Most table operations reach LanceDB through the LanceNamespace above, which is generated from the Lance Namespace specification. The MemWAL LSM routes are not part of that specification, so they are issued through a separate client:

import com.lancedb.LanceDbRestClient;
import com.lancedb.LanceDbTableLsm;
import com.lancedb.LsmWriteSpec;

LanceDbRestClient client = LanceDbNamespaceClientBuilder.newBuilder()
    .apiKey("your_lancedb_cloud_api_key")
    .database("your_database_name")
    .buildRestClient();

LanceDbTableLsm lsm = new LanceDbTableLsm(client, "my_table");

// Route future merge_insert upserts through the MemWAL, hash-bucketed by `id`.
lsm.setLsmWriteSpec(LsmWriteSpec.bucket("id", 16));

// ... merge_insert traffic ...

// Converge the fresh tier into the base table.
lsm.checkpointLsm();

// Inspect live per-bucket state.
lsm.getLsmStats().ifPresent(stats -> System.out.println(stats.get("buckets")));

client.close();

maintainedIndexes is tri-state, and the null default is the opposite of what a Java reader usually expects:

Value	Meaning
unset (null)	Maintain every index the MemWAL can, resolved on install
`Collections.emptyList()`	Maintain none
`Arrays.asList("id_idx")`	Maintain exactly those

Development

Build:

./mvnw install -pl lancedb-core -am

Run tests:

./mvnw test -pl lancedb-core

2.2 KiB Raw Permalink Blame History