mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-22 11:42:27 +00:00
Dan RammerandClaude Opus 5.5
92825f6f13
## What Renames the user-facing MemWAL surface to **RTE** (realtime engine) across Rust, Python, TypeScript, and Java, including the REST wire format. This follows the LSM → MemWAL rename in #4075. | Before | After | |---|---| | `MemWalWriteSpec` / `MemWalStats` | `RTEWriteSpec` / `RTEStats` | | `enable_memwal` / `disable_memwal` / `get_memwal_spec` | `enable_rte` / `disable_rte` / `get_rte_spec` | | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | `flush_rte` / `compact_rte` / `checkpoint_rte` | | `get_memwal_stats` / `close_memwal_writers` | `get_rte_stats` / `close_rte_writers` | | `use_memwal` (query, merge insert) | `use_rte` | | TS/Java `enableMemWal`, `useMemWal`, ... | `enableRTE`, `useRTE`, ... (napi `js_name` set explicitly) | | Java `LanceDbTableMemWal` | `LanceDbTableRTE` | | `table/memwal_stats.rs` | `table/rte_stats.rs` | Docs and comments now say "realtime engine (RTE)". Text about Lance's underlying mechanism keeps "MemWAL" ("Lance's MemWAL", "MemWAL index"). **Left alone:** Lance-owned names (`MemWalIndexDetails`, `DatasetMemWalExt`, `InitializeMemWal*`, `mem_wal` paths), internal `Lsm*` names and modules, and the `does not support with_row_id` refusal text that the hybrid fallback matches on. ## Compatibility Breaking, for an experimental API. The wire format changes, so this must roll out together with the matching sophon change (lancedb/sophon#8870): - Routes: `/enable_memwal/` → `/enable_rte/`, `/disable_memwal/` → `/disable_rte/`, `/get_memwal_spec/` → `/get_rte_spec/`, `/get_memwal_stats/` → `/get_rte_stats/`, `/flush_memwal/` → `/flush_rte/`, `/compact_memwal/` → `/compact_rte/` - Response keys: `memwal_spec` → `rte_spec`, `memwal_stats` → `rte_stats` - Request field: `use_memwal` → `use_rte` on query and merge-insert bodies The SQL keyword becomes `ALTER TABLE ... SET RTE` server-side. The guides under `docs/web` will be updated separately. ## Validation - Rust: `cargo clippy --features remote --tests --examples` clean; `cargo test --features remote -p lancedb --lib`: 1533 passed - Java: `mvn test`: 37 passed (Spotless applied with JDK 11) - Python: `ruff format` / `ruff check` clean; targeted tests (`test_rte_spec`, `test_merge_insert_lsm`, `test_remote_db`, `test_wal_hybrid`, `test_query`, `test_permutation`, `test_table`): 708 passed - TypeScript: `pnpm build`; `table`, `query` tests: 606 passed; `biome check` clean; `docs/src/js` regenerated with `pnpm run docs` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
92 lines
2.6 KiB
Markdown
92 lines
2.6 KiB
Markdown
# LanceDB Java Enterprise Client
|
|
|
|
## Configuration and Initialization
|
|
|
|
### LanceDB Cloud
|
|
|
|
For LanceDB Cloud, use the simplified builder API:
|
|
|
|
```java
|
|
import com.lancedb.LanceDbNamespaceClientBuilder;
|
|
import org.lance.namespace.LanceNamespace;
|
|
|
|
// If your DB url is db://example-db, then your database here is example-db
|
|
LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
|
.apiKey("your_lancedb_cloud_api_key")
|
|
.database("your_database_name")
|
|
.build();
|
|
```
|
|
|
|
### LanceDB Enterprise
|
|
|
|
For Enterprise deployments, use your custom endpoint:
|
|
|
|
```java
|
|
LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
|
.apiKey("your_lancedb_enterprise_api_key")
|
|
.database("your_database_name")
|
|
.endpoint("<your_enterprise_endpoint>")
|
|
.build();
|
|
```
|
|
|
|
## RTE write path
|
|
|
|
Most table operations reach LanceDB through the `LanceNamespace` above, which is
|
|
generated from the Lance Namespace specification. The RTE LSM routes are not part
|
|
of that specification, so they are issued through a separate client:
|
|
|
|
```java
|
|
import com.lancedb.LanceDbRestClient;
|
|
import com.lancedb.LanceDbTableRTE;
|
|
import com.lancedb.RTEWriteSpec;
|
|
|
|
LanceDbRestClient client = LanceDbNamespaceClientBuilder.newBuilder()
|
|
.apiKey("your_lancedb_cloud_api_key")
|
|
.database("your_database_name")
|
|
.buildRestClient();
|
|
|
|
LanceDbTableRTE rte = new LanceDbTableRTE(client, "my_table");
|
|
|
|
// Route future merge_insert upserts through the RTE, hash-bucketed by `id`.
|
|
rte.enableRTE(RTEWriteSpec.bucket("id", 16));
|
|
|
|
// ... merge_insert traffic ...
|
|
|
|
// Converge the fresh tier into the base table.
|
|
rte.checkpointRTE();
|
|
|
|
// Inspect live per-table-shard state.
|
|
rte.getRTEStats().ifPresent(stats -> stats.tableShards().forEach(tableShard ->
|
|
System.out.println(tableShard.shardId() + ": " + tableShard.sstables().size() + " SSTables")));
|
|
|
|
client.close();
|
|
```
|
|
|
|
`maintainedIndexes` is tri-state, and the null default is the opposite of what a Java
|
|
reader usually expects:
|
|
|
|
| Value | Meaning |
|
|
| --- | --- |
|
|
| unset (null) | Maintain **every** index the table has, including ones created later |
|
|
| `Collections.emptyList()` | Maintain **none** |
|
|
| `Arrays.asList("id_idx")` | Maintain exactly those |
|
|
|
|
## Development
|
|
|
|
The parent POM includes `https://maven.lance.org` to resolve Lance beta and RC
|
|
dependencies. Use an explicit `lance-core.version`; numbered beta and RC versions
|
|
are Maven releases, not snapshots. Maven Central remains available for stable
|
|
and historical versions, as well as other dependencies.
|
|
|
|
Build:
|
|
|
|
```shell
|
|
./mvnw install -pl lancedb-core -am
|
|
```
|
|
|
|
Run tests:
|
|
|
|
```shell
|
|
./mvnw test -pl lancedb-core
|
|
```
|