mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-10 07:12:30 +00:00
feat: infer maintained indexes when an LsmWriteSpec omits them (#3748)
## What `LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`: | value | meaning | |---|---| | `None` (new default) | every index the MemWAL supports, resolved when the spec is installed | | `Some([])` | maintain nothing — a scan/filter-only WAL table | | `Some([..])` | exactly these, taken verbatim | `with_maintained_indexes` keeps its signature; `with_no_maintained_indexes()` is new. Surfaced through the remote path (null on the wire), Python, and Node. ## Why Callers had to state the maintained set by hand every time, which is both tedious and easy to get wrong — the common case is "maintain what I already built." Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating to lance's `is_maintainable_index_type`. This is load-bearing rather than cosmetic: lance does **not** skip an index type its memtable cannot build, it errors when the shard writer opens, so sweeping up a bitmap index would fail every memtable claim and leave the table unwritable. The inferred set excludes those, and an explicit list naming one is now rejected at spec time instead of at claim time. ## Behavior change A freshly constructed spec used to maintain **nothing**; it now maintains **everything supported**. This flipped because napi collapses `undefined` and `null` to `None`, so TypeScript cannot express "absent means nothing, null means all" — any other choice makes the bindings disagree with the wire. The error direction also favors it: an unwanted maintained index costs memory, while a silently unmaintained one degrades FTS to an unscored scan. Three existing tests encoded the old default and are updated rather than worked around. ## Caveat The resolved set is a snapshot, not a subscription. An index created after the spec is installed is not maintained until the spec is unset and set again. `get_lsm_write_spec` therefore always reports a concrete list — `None` never round-trips. ## Dependency Needs a lance release carrying `is_maintainable_index_type` (lance-format/lance#8095) before this builds against the pinned tag. Draft until then. ## Testing 38 Rust LSM tests and 10 Python tests pass against a local lance build, including new coverage that a bitmap index is excluded from inference and rejected when named, and that `[]` stays distinguishable from null on the wire. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
be290447d9
commit
706a9c327f
@@ -83,7 +83,9 @@ def test_lsm_write_spec_repr():
|
||||
assert s.spec_type == "bucket"
|
||||
assert s.column == "id"
|
||||
assert s.num_buckets == 4
|
||||
assert s.maintained_indexes == []
|
||||
# A fresh spec defers its maintained set to install time.
|
||||
assert s.maintained_indexes is None
|
||||
assert s.with_maintained_indexes([]).maintained_indexes == []
|
||||
assert "bucket" in repr(s)
|
||||
assert "id" in repr(s)
|
||||
assert "4" in repr(s)
|
||||
@@ -169,18 +171,23 @@ def test_get_lsm_write_spec(tmp_path):
|
||||
table.unset_lsm_write_spec()
|
||||
assert table.get_lsm_write_spec() is None
|
||||
|
||||
# Identity round-trips (column recovered from the schema).
|
||||
# Identity round-trips (column recovered from the schema). Leaving the
|
||||
# maintained set to be inferred picks up the index on the table, so the
|
||||
# spec reads back naming it rather than as "infer".
|
||||
table.set_lsm_write_spec(LsmWriteSpec.identity("id"))
|
||||
spec = table.get_lsm_write_spec()
|
||||
assert spec.spec_type == "identity"
|
||||
assert spec.column == "id"
|
||||
assert spec.maintained_indexes == [idx_name]
|
||||
table.unset_lsm_write_spec()
|
||||
|
||||
# Unsharded round-trips (no routing column).
|
||||
table.set_lsm_write_spec(LsmWriteSpec.unsharded())
|
||||
# Unsharded round-trips (no routing column). Opting out is distinct from
|
||||
# the inferred default.
|
||||
table.set_lsm_write_spec(LsmWriteSpec.unsharded().with_maintained_indexes([]))
|
||||
spec = table.get_lsm_write_spec()
|
||||
assert spec.spec_type == "unsharded"
|
||||
assert spec.column is None
|
||||
assert spec.maintained_indexes == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
Reference in New Issue
Block a user