mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-28 06:26:15 +00:00
refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dan Rammer
co-authored by
Claude Opus 5
Claude Opus 5
parent
0968c21f4c
commit
fe34eb3549
@@ -1845,10 +1845,10 @@ def lsm_test_table(lsm_handler):
|
||||
"""A remote table whose LSM routes are served by ``lsm_handler``.
|
||||
|
||||
``lsm_handler(request, route)`` is called for ``/v1/table/test/<route>/``
|
||||
where route is one of flush_lsm, compact_lsm, get_lsm_stats, and is
|
||||
where route is one of flush_memwal, compact_memwal, get_memwal_stats, and is
|
||||
responsible for writing the response.
|
||||
"""
|
||||
routes = ("flush_lsm", "compact_lsm", "get_lsm_stats")
|
||||
routes = ("flush_memwal", "compact_memwal", "get_memwal_stats")
|
||||
|
||||
def handler(request):
|
||||
match = re.fullmatch(r"/v1/table/test/(\w+)/", request.path)
|
||||
@@ -1880,9 +1880,9 @@ def send_json(request, payload, status=200):
|
||||
request.wfile.write(json.dumps(payload).encode())
|
||||
|
||||
|
||||
def test_get_lsm_stats_sync():
|
||||
def test_get_memwal_stats_sync():
|
||||
"""The sync wrapper round-trips the server payload into a dict."""
|
||||
bucket = {
|
||||
table_shard = {
|
||||
"shard_id": "b0",
|
||||
"status": "Active",
|
||||
"writer_epoch": 3,
|
||||
@@ -1890,7 +1890,7 @@ def test_get_lsm_stats_sync():
|
||||
"current_generation": 6,
|
||||
"replay_after_wal_entry_position": 40,
|
||||
"wal_entry_position_last_seen": 42,
|
||||
"generations": [{"generation": 5, "bytes": 1024, "rows": 7}],
|
||||
"sstables": [{"generation": 5, "bytes": 1024, "rows": 7}],
|
||||
"compacting": False,
|
||||
"memtables": [
|
||||
{
|
||||
@@ -1905,29 +1905,29 @@ def test_get_lsm_stats_sync():
|
||||
seen_bodies = []
|
||||
|
||||
def lsm_handler(request, route):
|
||||
assert route == "get_lsm_stats"
|
||||
assert route == "get_memwal_stats"
|
||||
seen_bodies.append(read_json_body(request))
|
||||
send_json(request, {"lsm_stats": {"buckets": [bucket]}})
|
||||
send_json(request, {"memwal_stats": {"table_shards": [table_shard]}})
|
||||
|
||||
with lsm_test_table(lsm_handler) as table:
|
||||
assert table.get_lsm_stats() == {"buckets": [bucket]}
|
||||
assert table.get_memwal_stats() == {"table_shards": [table_shard]}
|
||||
# Off by default, and forwarded when asked for.
|
||||
assert seen_bodies == [{"include_generation_rows": False}]
|
||||
table.get_lsm_stats(include_generation_rows=True)
|
||||
assert seen_bodies[-1] == {"include_generation_rows": True}
|
||||
assert seen_bodies == [{"include_sstable_rows": False}]
|
||||
table.get_memwal_stats(include_sstable_rows=True)
|
||||
assert seen_bodies[-1] == {"include_sstable_rows": True}
|
||||
|
||||
|
||||
def test_get_lsm_stats_sync_returns_none_when_lsm_disabled():
|
||||
"""A null envelope means the LSM write path is not enabled, not an error."""
|
||||
def test_get_memwal_stats_sync_returns_none_when_memwal_disabled():
|
||||
"""A null envelope means the MemWAL write path is not enabled, not an error."""
|
||||
|
||||
def lsm_handler(request, route):
|
||||
send_json(request, {"lsm_stats": None})
|
||||
send_json(request, {"memwal_stats": None})
|
||||
|
||||
with lsm_test_table(lsm_handler) as table:
|
||||
assert table.get_lsm_stats() is None
|
||||
assert table.get_memwal_stats() is None
|
||||
|
||||
|
||||
def test_flush_and_compact_lsm_sync():
|
||||
def test_flush_and_compact_memwal_sync():
|
||||
"""Both are one-shot POSTs answered 202 with no body."""
|
||||
called = []
|
||||
|
||||
@@ -1937,13 +1937,13 @@ def test_flush_and_compact_lsm_sync():
|
||||
request.end_headers()
|
||||
|
||||
with lsm_test_table(lsm_handler) as table:
|
||||
assert table.flush_lsm() is None
|
||||
assert table.compact_lsm() is None
|
||||
assert called == ["flush_lsm", "compact_lsm"]
|
||||
assert table.flush_memwal() is None
|
||||
assert table.compact_memwal() is None
|
||||
assert called == ["flush_memwal", "compact_memwal"]
|
||||
|
||||
|
||||
def test_checkpoint_lsm_sync():
|
||||
"""Seal, read the watermark, and return once L0 holds nothing.
|
||||
def test_checkpoint_memwal_sync():
|
||||
"""Freeze, read the watermark, and return once no SSTables remain.
|
||||
|
||||
The convergence loop itself is covered in Rust; this pins the sync
|
||||
binding to the endpoints it drives.
|
||||
@@ -1952,17 +1952,17 @@ def test_checkpoint_lsm_sync():
|
||||
|
||||
def lsm_handler(request, route):
|
||||
called.append(route)
|
||||
if route == "get_lsm_stats":
|
||||
# An empty L0 yields no target watermark, so the loop is done
|
||||
# after the seal without ever polling compaction.
|
||||
send_json(request, {"lsm_stats": {"buckets": []}})
|
||||
if route == "get_memwal_stats":
|
||||
# An empty SSTable tier yields no target watermark, so the loop is done
|
||||
# after the freeze without ever polling compaction.
|
||||
send_json(request, {"memwal_stats": {"table_shards": []}})
|
||||
else:
|
||||
request.send_response(202)
|
||||
request.end_headers()
|
||||
|
||||
with lsm_test_table(lsm_handler) as table:
|
||||
assert table.checkpoint_lsm() is None
|
||||
assert called == ["flush_lsm", "get_lsm_stats"]
|
||||
assert table.checkpoint_memwal() is None
|
||||
assert called == ["flush_memwal", "get_memwal_stats"]
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
|
||||
Reference in New Issue
Block a user