feat: infer maintained indexes when an LsmWriteSpec omits them (#3748)

## What

`LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`:

| value | meaning |
|---|---|
| `None` (new default) | every index the MemWAL supports, resolved when
the spec is installed |
| `Some([])` | maintain nothing — a scan/filter-only WAL table |
| `Some([..])` | exactly these, taken verbatim |

`with_maintained_indexes` keeps its signature;
`with_no_maintained_indexes()` is new. Surfaced through the remote path
(null on the wire), Python, and Node.

## Why

Callers had to state the maintained set by hand every time, which is
both tedious and easy to get wrong — the common case is "maintain what I
already built."

Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating
to lance's `is_maintainable_index_type`. This is load-bearing rather
than cosmetic: lance does **not** skip an index type its memtable cannot
build, it errors when the shard writer opens, so sweeping up a bitmap
index would fail every memtable claim and leave the table unwritable.
The inferred set excludes those, and an explicit list naming one is now
rejected at spec time instead of at claim time.

## Behavior change

A freshly constructed spec used to maintain **nothing**; it now
maintains **everything supported**. This flipped because napi collapses
`undefined` and `null` to `None`, so TypeScript cannot express "absent
means nothing, null means all" — any other choice makes the bindings
disagree with the wire. The error direction also favors it: an unwanted
maintained index costs memory, while a silently unmaintained one
degrades FTS to an unscored scan.

Three existing tests encoded the old default and are updated rather than
worked around.

## Caveat

The resolved set is a snapshot, not a subscription. An index created
after the spec is installed is not maintained until the spec is unset
and set again. `get_lsm_write_spec` therefore always reports a concrete
list — `None` never round-trips.

## Dependency

Needs a lance release carrying `is_maintainable_index_type`
(lance-format/lance#8095) before this builds against the pinned tag.
Draft until then.

## Testing

38 Rust LSM tests and 10 Python tests pass against a local lance build,
including new coverage that a bitmap index is excluded from inference
and rejected when named, and that `[]` stays distinguishable from null
on the wire.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dan Rammer
2026-08-07 14:50:22 -05:00
committed by GitHub
parent be290447d9
commit 706a9c327f
13 changed files with 360 additions and 93 deletions
+31 -15
View File
@@ -246,12 +246,22 @@ impl From<lancedb::table::MergeResult> for MergeResult {
}
}
/// Render for `__repr__`, so the default reads as Python's `None` rather than
/// Rust's `Some([..])`.
fn fmt_maintained(maintained: &Option<Vec<String>>) -> String {
match maintained {
Some(names) => format!("{:?}", names),
None => "None".to_string(),
}
}
/// Specification selecting Lance's MemWAL LSM-style write path for
/// `merge_insert`.
///
/// Constructed via the `bucket(...)`, `identity(...)`, or `unsharded()`
/// classmethods, then optionally chain `with_maintained_indexes(...)` and
/// `with_writer_config_defaults(...)`.
/// `with_writer_config_defaults(...)`. A fresh spec maintains every index the
/// MemWAL supports, resolved on install.
#[pyclass(from_py_object)]
#[derive(Clone, Debug)]
pub struct LsmWriteSpec {
@@ -291,11 +301,11 @@ impl LsmWriteSpec {
}
}
/// Replace the list of indexes the MemWAL should keep up to date as
/// rows are appended. Each name must reference an index that
/// already exists on the table at the time `set_lsm_write_spec`
/// is called.
pub fn with_maintained_indexes(&self, indexes: Vec<String>) -> Self {
/// Set which indexes the MemWAL maintains. `None` (the default)
/// resolves every supported index on install; a list is verbatim,
/// and an empty list maintains nothing.
#[pyo3(signature = (indexes))]
pub fn with_maintained_indexes(&self, indexes: Option<Vec<String>>) -> Self {
Self {
inner: self.inner.clone().with_maintained_indexes(indexes),
}
@@ -317,23 +327,29 @@ impl LsmWriteSpec {
maintained_indexes,
writer_config_defaults,
} => format!(
"LsmWriteSpec.bucket(column={:?}, num_buckets={}, maintained_indexes={:?}, writer_config_defaults={:?})",
column, num_buckets, maintained_indexes, writer_config_defaults,
"LsmWriteSpec.bucket(column={:?}, num_buckets={}, maintained_indexes={}, writer_config_defaults={:?})",
column,
num_buckets,
fmt_maintained(maintained_indexes),
writer_config_defaults,
),
lancedb::table::LsmWriteSpec::Identity {
column,
maintained_indexes,
writer_config_defaults,
} => format!(
"LsmWriteSpec.identity(column={:?}, maintained_indexes={:?}, writer_config_defaults={:?})",
column, maintained_indexes, writer_config_defaults,
"LsmWriteSpec.identity(column={:?}, maintained_indexes={}, writer_config_defaults={:?})",
column,
fmt_maintained(maintained_indexes),
writer_config_defaults,
),
lancedb::table::LsmWriteSpec::Unsharded {
maintained_indexes,
writer_config_defaults,
} => format!(
"LsmWriteSpec.unsharded(maintained_indexes={:?}, writer_config_defaults={:?})",
maintained_indexes, writer_config_defaults,
"LsmWriteSpec.unsharded(maintained_indexes={}, writer_config_defaults={:?})",
fmt_maintained(maintained_indexes),
writer_config_defaults,
),
}
}
@@ -368,10 +384,10 @@ impl LsmWriteSpec {
}
}
/// Names of indexes the MemWAL should keep up to date during writes.
/// Indexes the MemWAL keeps up to date, or `None` for every supported one.
#[getter]
pub fn maintained_indexes(&self) -> Vec<String> {
self.inner.maintained_indexes().to_vec()
pub fn maintained_indexes(&self) -> Option<Vec<String>> {
self.inner.maintained_indexes().map(<[String]>::to_vec)
}
/// Default `ShardWriter` configuration recorded by this spec.