Dan Rammer
706a9c327f
## What `LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`: | value | meaning | |---|---| | `None` (new default) | every index the MemWAL supports, resolved when the spec is installed | | `Some([])` | maintain nothing — a scan/filter-only WAL table | | `Some([..])` | exactly these, taken verbatim | `with_maintained_indexes` keeps its signature; `with_no_maintained_indexes()` is new. Surfaced through the remote path (null on the wire), Python, and Node. ## Why Callers had to state the maintained set by hand every time, which is both tedious and easy to get wrong — the common case is "maintain what I already built." Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating to lance's `is_maintainable_index_type`. This is load-bearing rather than cosmetic: lance does **not** skip an index type its memtable cannot build, it errors when the shard writer opens, so sweeping up a bitmap index would fail every memtable claim and leave the table unwritable. The inferred set excludes those, and an explicit list naming one is now rejected at spec time instead of at claim time. ## Behavior change A freshly constructed spec used to maintain **nothing**; it now maintains **everything supported**. This flipped because napi collapses `undefined` and `null` to `None`, so TypeScript cannot express "absent means nothing, null means all" — any other choice makes the bindings disagree with the wire. The error direction also favors it: an unwanted maintained index costs memory, while a silently unmaintained one degrades FTS to an unscored scan. Three existing tests encoded the old default and are updated rather than worked around. ## Caveat The resolved set is a snapshot, not a subscription. An index created after the spec is installed is not maintained until the spec is unset and set again. `get_lsm_write_spec` therefore always reports a concrete list — `None` never round-trips. ## Dependency Needs a lance release carrying `is_maintainable_index_type` (lance-format/lance#8095) before this builds against the pinned tag. Draft until then. ## Testing 38 Rust LSM tests and 10 Python tests pass against a local lance build, including new coverage that a bitmap index is excluded from inference and rejected when named, and that `[]` stays distinguishable from null on the wire. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1.3 KiB
@lancedb/lancedb • Docs
@lancedb/lancedb / LsmWriteSpec
Interface: LsmWriteSpec
Specification selecting Lance's MemWAL LSM-style write path for
mergeInsert.
specType is "bucket", "identity", or "unsharded". For "bucket",
column and numBuckets are required; for "identity", column is
required and must be a deterministic function of the unenforced primary
key (every row with a given primary key must always produce the same
column value, or upserts of that key can land in different shards and a
stale version can win).
Properties
column?
optional column: string;
Bucket and identity variants: the sharding column.
maintainedIndexes?
optional maintainedIndexes: string[];
Indexes the MemWAL keeps up to date. Omit to maintain every supported
index, resolved on install — a snapshot, so indexes created later are not
maintained. Pass [] for none.
numBuckets?
optional numBuckets: number;
Bucket variant: the number of buckets, in [1, 1024].
specType
specType: "bucket" | "identity" | "unsharded";
One of "bucket", "identity", or "unsharded".
writerConfigDefaults?
optional writerConfigDefaults: Record<string, string>;
Default ShardWriter configuration recorded in the MemWAL index.