mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-23 04:02:27 +00:00
refactor!: rename MemWAL APIs to RTE (realtime engine) (#4598)
## What Renames the user-facing MemWAL surface to **RTE** (realtime engine) across Rust, Python, TypeScript, and Java, including the REST wire format. This follows the LSM → MemWAL rename in #4075. | Before | After | |---|---| | `MemWalWriteSpec` / `MemWalStats` | `RTEWriteSpec` / `RTEStats` | | `enable_memwal` / `disable_memwal` / `get_memwal_spec` | `enable_rte` / `disable_rte` / `get_rte_spec` | | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | `flush_rte` / `compact_rte` / `checkpoint_rte` | | `get_memwal_stats` / `close_memwal_writers` | `get_rte_stats` / `close_rte_writers` | | `use_memwal` (query, merge insert) | `use_rte` | | TS/Java `enableMemWal`, `useMemWal`, ... | `enableRTE`, `useRTE`, ... (napi `js_name` set explicitly) | | Java `LanceDbTableMemWal` | `LanceDbTableRTE` | | `table/memwal_stats.rs` | `table/rte_stats.rs` | Docs and comments now say "realtime engine (RTE)". Text about Lance's underlying mechanism keeps "MemWAL" ("Lance's MemWAL", "MemWAL index"). **Left alone:** Lance-owned names (`MemWalIndexDetails`, `DatasetMemWalExt`, `InitializeMemWal*`, `mem_wal` paths), internal `Lsm*` names and modules, and the `does not support with_row_id` refusal text that the hybrid fallback matches on. ## Compatibility Breaking, for an experimental API. The wire format changes, so this must roll out together with the matching sophon change (lancedb/sophon#8870): - Routes: `/enable_memwal/` → `/enable_rte/`, `/disable_memwal/` → `/disable_rte/`, `/get_memwal_spec/` → `/get_rte_spec/`, `/get_memwal_stats/` → `/get_rte_stats/`, `/flush_memwal/` → `/flush_rte/`, `/compact_memwal/` → `/compact_rte/` - Response keys: `memwal_spec` → `rte_spec`, `memwal_stats` → `rte_stats` - Request field: `use_memwal` → `use_rte` on query and merge-insert bodies The SQL keyword becomes `ALTER TABLE ... SET RTE` server-side. The guides under `docs/web` will be updated separately. ## Validation - Rust: `cargo clippy --features remote --tests --examples` clean; `cargo test --features remote -p lancedb --lib`: 1533 passed - Java: `mvn test`: 37 passed (Spotless applied with JDK 11) - Python: `ruff format` / `ruff check` clean; targeted tests (`test_rte_spec`, `test_merge_insert_lsm`, `test_remote_db`, `test_wal_hybrid`, `test_query`, `test_permutation`, `test_table`): 708 passed - TypeScript: `pnpm build`; `table`, `query` tests: 606 passed; `biome check` clean; `docs/src/js` regenerated with `pnpm run docs` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
1772f4ef83
commit
92825f6f13
@@ -659,7 +659,7 @@ minimum; one searches all partitions within the caller maximum. Later
|
||||
probing may continue when too few candidates are found.
|
||||
Any explicit effort, including 0.5, overrides nprobes in either call order.
|
||||
Explicit minimum and maximum bounds still apply. This is not a recall
|
||||
guarantee and does not change refinement or HNSW ef. The MemWAL LSM
|
||||
guarantee and does not change refinement or HNSW ef. The RTE LSM
|
||||
scanner does not yet support this option.
|
||||
|
||||
#### Parameters
|
||||
@@ -781,16 +781,16 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useMemWal()
|
||||
### useRTE()
|
||||
|
||||
```ts
|
||||
useMemWal(enable): this
|
||||
useRTE(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this query.
|
||||
Control RTE read routing for this query.
|
||||
|
||||
By default (unset), when the table carries a MemWAL write spec (see
|
||||
[Table#enableMemWal](Table.md#enablememwal)), reads are routed through the LSM scanner so
|
||||
By default (unset), when the table carries an RTE write spec (see
|
||||
[Table#enableRTE](Table.md#enablerte)), reads are routed through the LSM scanner so
|
||||
they also return data written via the `mergeInsert` LSM path that has not yet
|
||||
been compacted into the base table (the active/frozen in-memory memtables and
|
||||
the SSTables), deduplicated by primary key; a table without a spec
|
||||
@@ -800,12 +800,12 @@ reads the base table.
|
||||
|
||||
* **enable**: `boolean`
|
||||
`true` forces the LSM scanner and errors if the table has no
|
||||
MemWAL write spec. `false` bypasses the MemWAL and reads the base table only,
|
||||
RTE write spec. `false` bypasses the RTE and reads the base table only,
|
||||
even when a spec is present.
|
||||
Note: the LSM scanner does not support every query shape (e.g. reranking,
|
||||
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
|
||||
`useMemWal(false)` is set, because a base-only read would silently exclude
|
||||
un-compacted MemWAL data.
|
||||
hybrid search, `orderBy`). On an RTE table those shapes error unless
|
||||
`useRTE(false)` is set, because a base-only read would silently exclude
|
||||
un-compacted RTE data.
|
||||
|
||||
#### Returns
|
||||
|
||||
@@ -813,7 +813,7 @@ reads the base table.
|
||||
|
||||
#### Inherited from
|
||||
|
||||
`StandardQueryBase.useMemWal`
|
||||
`StandardQueryBase.useRTE`
|
||||
|
||||
***
|
||||
|
||||
|
||||
Reference in New Issue
Block a user