mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-22 03:32:28 +00:00
refactor!: rename MemWAL APIs to RTE (realtime engine) (#4598)
## What Renames the user-facing MemWAL surface to **RTE** (realtime engine) across Rust, Python, TypeScript, and Java, including the REST wire format. This follows the LSM → MemWAL rename in #4075. | Before | After | |---|---| | `MemWalWriteSpec` / `MemWalStats` | `RTEWriteSpec` / `RTEStats` | | `enable_memwal` / `disable_memwal` / `get_memwal_spec` | `enable_rte` / `disable_rte` / `get_rte_spec` | | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | `flush_rte` / `compact_rte` / `checkpoint_rte` | | `get_memwal_stats` / `close_memwal_writers` | `get_rte_stats` / `close_rte_writers` | | `use_memwal` (query, merge insert) | `use_rte` | | TS/Java `enableMemWal`, `useMemWal`, ... | `enableRTE`, `useRTE`, ... (napi `js_name` set explicitly) | | Java `LanceDbTableMemWal` | `LanceDbTableRTE` | | `table/memwal_stats.rs` | `table/rte_stats.rs` | Docs and comments now say "realtime engine (RTE)". Text about Lance's underlying mechanism keeps "MemWAL" ("Lance's MemWAL", "MemWAL index"). **Left alone:** Lance-owned names (`MemWalIndexDetails`, `DatasetMemWalExt`, `InitializeMemWal*`, `mem_wal` paths), internal `Lsm*` names and modules, and the `does not support with_row_id` refusal text that the hybrid fallback matches on. ## Compatibility Breaking, for an experimental API. The wire format changes, so this must roll out together with the matching sophon change (lancedb/sophon#8870): - Routes: `/enable_memwal/` → `/enable_rte/`, `/disable_memwal/` → `/disable_rte/`, `/get_memwal_spec/` → `/get_rte_spec/`, `/get_memwal_stats/` → `/get_rte_stats/`, `/flush_memwal/` → `/flush_rte/`, `/compact_memwal/` → `/compact_rte/` - Response keys: `memwal_spec` → `rte_spec`, `memwal_stats` → `rte_stats` - Request field: `use_memwal` → `use_rte` on query and merge-insert bodies The SQL keyword becomes `ALTER TABLE ... SET RTE` server-side. The guides under `docs/web` will be updated separately. ## Validation - Rust: `cargo clippy --features remote --tests --examples` clean; `cargo test --features remote -p lancedb --lib`: 1533 passed - Java: `mvn test`: 37 passed (Spotless applied with JDK 11) - Python: `ruff format` / `ruff check` clean; targeted tests (`test_rte_spec`, `test_merge_insert_lsm`, `test_remote_db`, `test_wal_hybrid`, `test_query`, `test_permutation`, `test_table`): 708 passed - TypeScript: `pnpm build`; `table`, `query` tests: 606 passed; `biome check` clean; `docs/src/js` regenerated with `pnpm run docs` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
1772f4ef83
commit
92825f6f13
+9
-9
@@ -29,34 +29,34 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.build();
|
||||
```
|
||||
|
||||
## MemWAL write path
|
||||
## RTE write path
|
||||
|
||||
Most table operations reach LanceDB through the `LanceNamespace` above, which is
|
||||
generated from the Lance Namespace specification. The MemWAL LSM routes are not part
|
||||
generated from the Lance Namespace specification. The RTE LSM routes are not part
|
||||
of that specification, so they are issued through a separate client:
|
||||
|
||||
```java
|
||||
import com.lancedb.LanceDbRestClient;
|
||||
import com.lancedb.LanceDbTableMemWal;
|
||||
import com.lancedb.MemWalWriteSpec;
|
||||
import com.lancedb.LanceDbTableRTE;
|
||||
import com.lancedb.RTEWriteSpec;
|
||||
|
||||
LanceDbRestClient client = LanceDbNamespaceClientBuilder.newBuilder()
|
||||
.apiKey("your_lancedb_cloud_api_key")
|
||||
.database("your_database_name")
|
||||
.buildRestClient();
|
||||
|
||||
LanceDbTableMemWal memWal = new LanceDbTableMemWal(client, "my_table");
|
||||
LanceDbTableRTE rte = new LanceDbTableRTE(client, "my_table");
|
||||
|
||||
// Route future merge_insert upserts through the MemWAL, hash-bucketed by `id`.
|
||||
memWal.enableMemWal(MemWalWriteSpec.bucket("id", 16));
|
||||
// Route future merge_insert upserts through the RTE, hash-bucketed by `id`.
|
||||
rte.enableRTE(RTEWriteSpec.bucket("id", 16));
|
||||
|
||||
// ... merge_insert traffic ...
|
||||
|
||||
// Converge the fresh tier into the base table.
|
||||
memWal.checkpointMemWal();
|
||||
rte.checkpointRTE();
|
||||
|
||||
// Inspect live per-table-shard state.
|
||||
memWal.getMemWalStats().ifPresent(stats -> stats.tableShards().forEach(tableShard ->
|
||||
rte.getRTEStats().ifPresent(stats -> stats.tableShards().forEach(tableShard ->
|
||||
System.out.println(tableShard.shardId() + ": " + tableShard.sstables().size() + " SSTables")));
|
||||
|
||||
client.close();
|
||||
|
||||
Reference in New Issue
Block a user